kubernetes-sigs/azuredisk-csi-driver PR #1643 pull-azuredisk-csi-driver-e2e-multi-az #1605759807774527488
Recent runs || View in Spyglass
Recent runs || View in Spyglass
| PR | andyzhangx: feat: add disable-update-cache flag |
| Result | FAILURE |
| Tests | 2 failed / 38 succeeded |
| Started | |
| Elapsed | 1h19m |
| Revision | 6256c498f32db3e7ba9825ec99a29eb39c34e056 |
| Refs |
1643 |
| job-version | v1.27.0-alpha.0.588+d2504c94a0116d |
| kubetest-version | v20221221-461b6105cf |
| revision | v1.27.0-alpha.0.588+d2504c94a0116d |
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=AzureDisk\sCSI\sDriver\sEnd\-to\-End\sTests\sDynamic\sProvisioning\s\[multi\-az\]\sshould\screate\sa\spod\,\swrite\sto\sits\spv\,\stake\sa\svolume\ssnapshot\,\soverwrite\sdata\sin\soriginal\spv\,\screate\sanother\spod\sfrom\sthe\ssnapshot\,\sand\sread\sunaltered\soriginal\sdata\sfrom\soriginal\spv\[disk\.csi\.azure\.com\]$'
/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:745
Unexpected error:
<*errors.errorString | 0xc000693450>: {
s: "pod \"azuredisk-volume-tester-vdjl5\" failed with status: {Phase:Failed Conditions:[{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-12-22 03:52:00 +0000 UTC Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-12-22 03:52:00 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [volume-tester]} {Type:ContainersReady Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-12-22 03:52:00 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [volume-tester]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-12-22 03:52:00 +0000 UTC Reason: Message:}] Message: Reason: NominatedNodeName: HostIP:10.248.0.32 PodIP:10.248.0.47 PodIPs:[{IP:10.248.0.47}] StartTime:2022-12-22 03:52:00 +0000 UTC InitContainerStatuses:[] ContainerStatuses:[{Name:volume-tester State:{Waiting:nil Running:nil Terminated:&ContainerStateTerminated{ExitCode:2,Signal:0,Reason:Error,Message:,StartedAt:2022-12-22 03:52:19 +0000 UTC,FinishedAt:2022-12-22 03:52:19 +0000 UTC,ContainerID:containerd://b2c1c4ab10cb869d544b2e41ef8fa99ea0008a8f043d604ff3784c8b1e687a43,}} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:false RestartCount:0 Image:k8s.gcr.io/e2e-test-images/busybox:1.29-2 ImageID:k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf ContainerID:containerd://b2c1c4ab10cb869d544b2e41ef8fa99ea0008a8f043d604ff3784c8b1e687a43 Started:0xc000520e90}] QOSClass:BestEffort EphemeralContainerStatuses:[]}",
}
pod "azuredisk-volume-tester-vdjl5" failed with status: {Phase:Failed Conditions:[{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-12-22 03:52:00 +0000 UTC Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-12-22 03:52:00 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [volume-tester]} {Type:ContainersReady Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-12-22 03:52:00 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [volume-tester]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-12-22 03:52:00 +0000 UTC Reason: Message:}] Message: Reason: NominatedNodeName: HostIP:10.248.0.32 PodIP:10.248.0.47 PodIPs:[{IP:10.248.0.47}] StartTime:2022-12-22 03:52:00 +0000 UTC InitContainerStatuses:[] ContainerStatuses:[{Name:volume-tester State:{Waiting:nil Running:nil Terminated:&ContainerStateTerminated{ExitCode:2,Signal:0,Reason:Error,Message:,StartedAt:2022-12-22 03:52:19 +0000 UTC,FinishedAt:2022-12-22 03:52:19 +0000 UTC,ContainerID:containerd://b2c1c4ab10cb869d544b2e41ef8fa99ea0008a8f043d604ff3784c8b1e687a43,}} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:false RestartCount:0 Image:k8s.gcr.io/e2e-test-images/busybox:1.29-2 ImageID:k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf ContainerID:containerd://b2c1c4ab10cb869d544b2e41ef8fa99ea0008a8f043d604ff3784c8b1e687a43 Started:0xc000520e90}] QOSClass:BestEffort EphemeralContainerStatuses:[]}
occurred
/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/testsuites/testsuites.go:825
�[1mSTEP�[0m: Creating a kubernetes client
Dec 22 03:51:15.172: INFO: >>> kubeConfig: /root/tmp1530367720/kubeconfig/kubeconfig.westus2.json
�[1mSTEP�[0m: Building a namespace api object, basename azuredisk
�[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace
�[1mSTEP�[0m: Waiting for kube-root-ca.crt to be provisioned in namespace
Dec 22 03:51:16.063: INFO: >>> kubeConfig: /root/tmp1530367720/kubeconfig/kubeconfig.westus2.json
�[1mSTEP�[0m: setting up the StorageClass
�[1mSTEP�[0m: creating a StorageClass
�[1mSTEP�[0m: setting up the PVC and PV
�[1mSTEP�[0m: creating a PVC
�[1mSTEP�[0m: deploying the pod
�[1mSTEP�[0m: checking that the pod's command exits with no error
Dec 22 03:51:16.252: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-fdmqt" in namespace "azuredisk-7726" to be "Succeeded or Failed"
Dec 22 03:51:16.308: INFO: Pod "azuredisk-volume-tester-fdmqt": Phase="Pending", Reason="", readiness=false. Elapsed: 56.868349ms
Dec 22 03:51:18.368: INFO: Pod "azuredisk-volume-tester-fdmqt": Phase="Pending", Reason="", readiness=false. Elapsed: 2.116528825s
Dec 22 03:51:20.431: INFO: Pod "azuredisk-volume-tester-fdmqt": Phase="Pending", Reason="", readiness=false. Elapsed: 4.179142357s
Dec 22 03:51:22.491: INFO: Pod "azuredisk-volume-tester-fdmqt": Phase="Pending", Reason="", readiness=false. Elapsed: 6.239349476s
Dec 22 03:51:24.551: INFO: Pod "azuredisk-volume-tester-fdmqt": Phase="Pending", Reason="", readiness=false. Elapsed: 8.299042471s
Dec 22 03:51:26.612: INFO: Pod "azuredisk-volume-tester-fdmqt": Phase="Pending", Reason="", readiness=false. Elapsed: 10.36010098s
Dec 22 03:51:28.672: INFO: Pod "azuredisk-volume-tester-fdmqt": Phase="Pending", Reason="", readiness=false. Elapsed: 12.420498815s
Dec 22 03:51:30.730: INFO: Pod "azuredisk-volume-tester-fdmqt": Phase="Pending", Reason="", readiness=false. Elapsed: 14.478923369s
Dec 22 03:51:32.791: INFO: Pod "azuredisk-volume-tester-fdmqt": Phase="Pending", Reason="", readiness=false. Elapsed: 16.539061812s
Dec 22 03:51:34.851: INFO: Pod "azuredisk-volume-tester-fdmqt": Phase="Pending", Reason="", readiness=false. Elapsed: 18.599768938s
Dec 22 03:51:36.910: INFO: Pod "azuredisk-volume-tester-fdmqt": Phase="Pending", Reason="", readiness=false. Elapsed: 20.658776422s
Dec 22 03:51:38.969: INFO: Pod "azuredisk-volume-tester-fdmqt": Phase="Succeeded", Reason="", readiness=false. Elapsed: 22.717766217s
�[1mSTEP�[0m: Saw pod success
Dec 22 03:51:38.969: INFO: Pod "azuredisk-volume-tester-fdmqt" satisfied condition "Succeeded or Failed"
�[1mSTEP�[0m: Checking Prow test resource group
�[1mSTEP�[0m: Prow test resource group: kubetest-cg1hzjpb
�[1mSTEP�[0m: Creating external resource group: azuredisk-csi-driver-test-f052e28a-81ab-11ed-84f5-fa58d3d99fda
�[1mSTEP�[0m: creating volume snapshot class with external rg azuredisk-csi-driver-test-f052e28a-81ab-11ed-84f5-fa58d3d99fda
�[1mSTEP�[0m: setting up the VolumeSnapshotClass
�[1mSTEP�[0m: creating a VolumeSnapshotClass
�[1mSTEP�[0m: taking snapshots
�[1mSTEP�[0m: creating a VolumeSnapshot for pvc-f46kx
�[1mSTEP�[0m: deploying a new pod to overwrite pv data
�[1mSTEP�[0m: checking that the pod is running
�[1mSTEP�[0m: waiting for VolumeSnapshot to be ready to use - volume-snapshot-kgkg4
�[1mSTEP�[0m: setting up the StorageClass
�[1mSTEP�[0m: creating a StorageClass
�[1mSTEP�[0m: setting up the PVC and PV
�[1mSTEP�[0m: creating a PVC
�[1mSTEP�[0m: setting up the pod
�[1mSTEP�[0m: Set pod anti-affinity to make sure two pods are scheduled on different nodes
�[1mSTEP�[0m: deploying a pod with a volume restored from the snapshot
�[1mSTEP�[0m: checking that the pod's command exits with no error
Dec 22 03:51:57.307: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-vdjl5" in namespace "azuredisk-7726" to be "Succeeded or Failed"
Dec 22 03:51:57.364: INFO: Pod "azuredisk-volume-tester-vdjl5": Phase="Pending", Reason="", readiness=false. Elapsed: 57.274216ms
Dec 22 03:51:59.424: INFO: Pod "azuredisk-volume-tester-vdjl5": Phase="Pending", Reason="", readiness=false. Elapsed: 2.116525378s
Dec 22 03:52:01.483: INFO: Pod "azuredisk-volume-tester-vdjl5": Phase="Pending", Reason="", readiness=false. Elapsed: 4.175913955s
Dec 22 03:52:03.543: INFO: Pod "azuredisk-volume-tester-vdjl5": Phase="Pending", Reason="", readiness=false. Elapsed: 6.235524118s
Dec 22 03:52:05.602: INFO: Pod "azuredisk-volume-tester-vdjl5": Phase="Pending", Reason="", readiness=false. Elapsed: 8.294998803s
Dec 22 03:52:07.662: INFO: Pod "azuredisk-volume-tester-vdjl5": Phase="Pending", Reason="", readiness=false. Elapsed: 10.354554016s
Dec 22 03:52:09.720: INFO: Pod "azuredisk-volume-tester-vdjl5": Phase="Pending", Reason="", readiness=false. Elapsed: 12.413268115s
Dec 22 03:52:11.780: INFO: Pod "azuredisk-volume-tester-vdjl5": Phase="Pending", Reason="", readiness=false. Elapsed: 14.473089312s
Dec 22 03:52:13.840: INFO: Pod "azuredisk-volume-tester-vdjl5": Phase="Pending", Reason="", readiness=false. Elapsed: 16.533061094s
Dec 22 03:52:15.899: INFO: Pod "azuredisk-volume-tester-vdjl5": Phase="Pending", Reason="", readiness=false. Elapsed: 18.592086654s
Dec 22 03:52:17.958: INFO: Pod "azuredisk-volume-tester-vdjl5": Phase="Pending", Reason="", readiness=false. Elapsed: 20.651267428s
Dec 22 03:52:20.018: INFO: Pod "azuredisk-volume-tester-vdjl5": Phase="Failed", Reason="", readiness=false. Elapsed: 22.710872746s
Dec 22 03:52:20.019: INFO: deleting Pod "azuredisk-7726"/"azuredisk-volume-tester-vdjl5"
Dec 22 03:52:20.111: INFO: Pod azuredisk-volume-tester-vdjl5 has the following logs: grep: /mnt/test-1/data: No such file or directory
�[1mSTEP�[0m: Deleting pod azuredisk-volume-tester-vdjl5 in namespace azuredisk-7726
�[1mSTEP�[0m: validating provisioned PV
�[1mSTEP�[0m: checking the PV
Dec 22 03:52:20.291: INFO: deleting PVC "azuredisk-7726"/"pvc-2qvds"
Dec 22 03:52:20.291: INFO: Deleting PersistentVolumeClaim "pvc-2qvds"
�[1mSTEP�[0m: waiting for claim's PV "pvc-88a56666-9755-42d9-84c7-fc664c7c98a4" to be deleted
Dec 22 03:52:20.354: INFO: Waiting up to 10m0s for PersistentVolume pvc-88a56666-9755-42d9-84c7-fc664c7c98a4 to get deleted
Dec 22 03:52:20.411: INFO: PersistentVolume pvc-88a56666-9755-42d9-84c7-fc664c7c98a4 found and phase=Released (57.493705ms)
Dec 22 03:52:25.474: INFO: PersistentVolume pvc-88a56666-9755-42d9-84c7-fc664c7c98a4 found and phase=Released (5.120111579s)
Dec 22 03:52:30.532: INFO: PersistentVolume pvc-88a56666-9755-42d9-84c7-fc664c7c98a4 found and phase=Released (10.178318675s)
Dec 22 03:52:35.594: INFO: PersistentVolume pvc-88a56666-9755-42d9-84c7-fc664c7c98a4 found and phase=Released (15.240739955s)
Dec 22 03:52:40.657: INFO: PersistentVolume pvc-88a56666-9755-42d9-84c7-fc664c7c98a4 found and phase=Released (20.303142183s)
Dec 22 03:52:45.717: INFO: PersistentVolume pvc-88a56666-9755-42d9-84c7-fc664c7c98a4 found and phase=Released (25.362961148s)
Dec 22 03:52:50.778: INFO: PersistentVolume pvc-88a56666-9755-42d9-84c7-fc664c7c98a4 found and phase=Released (30.424222274s)
Dec 22 03:52:55.842: INFO: PersistentVolume pvc-88a56666-9755-42d9-84c7-fc664c7c98a4 found and phase=Released (35.487840219s)
Dec 22 03:53:00.904: INFO: PersistentVolume pvc-88a56666-9755-42d9-84c7-fc664c7c98a4 was removed
Dec 22 03:53:00.904: INFO: Waiting up to 5m0s for PersistentVolumeClaim azuredisk-7726 to be removed
Dec 22 03:53:00.961: INFO: Claim "azuredisk-7726" in namespace "pvc-2qvds" doesn't exist in the system
Dec 22 03:53:00.961: INFO: deleting StorageClass azuredisk-7726-disk.csi.azure.com-dynamic-sc-w5ks2
�[1mSTEP�[0m: deleting a VolumeSnapshot volume-snapshot-kgkg4
Dec 22 03:53:01.098: INFO: deleting Pod "azuredisk-7726"/"azuredisk-volume-tester-q7z48"
Dec 22 03:53:01.182: INFO: Pod azuredisk-volume-tester-q7z48 has the following logs:
�[1mSTEP�[0m: Deleting pod azuredisk-volume-tester-q7z48 in namespace azuredisk-7726
Dec 22 03:53:01.244: INFO: skip deleting VolumeSnapshotClass azuredisk-7726-disk.csi.azure.com-dynamic-sc-xtnsx
Dec 22 03:53:01.244: INFO: Deleting resource group azuredisk-csi-driver-test-f052e28a-81ab-11ed-84f5-fa58d3d99fda
Dec 22 03:54:17.903: INFO: deleting Pod "azuredisk-7726"/"azuredisk-volume-tester-fdmqt"
Dec 22 03:54:17.965: INFO: Pod azuredisk-volume-tester-fdmqt has the following logs:
�[1mSTEP�[0m: Deleting pod azuredisk-volume-tester-fdmqt in namespace azuredisk-7726
�[1mSTEP�[0m: validating provisioned PV
�[1mSTEP�[0m: checking the PV
Dec 22 03:54:18.155: INFO: deleting PVC "azuredisk-7726"/"pvc-f46kx"
Dec 22 03:54:18.155: INFO: Deleting PersistentVolumeClaim "pvc-f46kx"
�[1mSTEP�[0m: waiting for claim's PV "pvc-f0eb7302-e84a-46b1-a371-062ff124d19f" to be deleted
Dec 22 03:54:18.217: INFO: Waiting up to 10m0s for PersistentVolume pvc-f0eb7302-e84a-46b1-a371-062ff124d19f to get deleted
Dec 22 03:54:18.275: INFO: PersistentVolume pvc-f0eb7302-e84a-46b1-a371-062ff124d19f found and phase=Released (57.802442ms)
Dec 22 03:54:23.338: INFO: PersistentVolume pvc-f0eb7302-e84a-46b1-a371-062ff124d19f found and phase=Released (5.120985464s)
Dec 22 03:54:28.397: INFO: PersistentVolume pvc-f0eb7302-e84a-46b1-a371-062ff124d19f was removed
Dec 22 03:54:28.397: INFO: Waiting up to 5m0s for PersistentVolumeClaim azuredisk-7726 to be removed
Dec 22 03:54:28.456: INFO: Claim "azuredisk-7726" in namespace "pvc-f46kx" doesn't exist in the system
Dec 22 03:54:28.456: INFO: deleting StorageClass azuredisk-7726-disk.csi.azure.com-dynamic-sc-q86z5
�[1mSTEP�[0m: Collecting events from namespace "azuredisk-7726".
�[1mSTEP�[0m: Found 30 events.
Dec 22 03:54:28.633: INFO: At 2022-12-22 03:51:16 +0000 UTC - event for pvc-f46kx: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
Dec 22 03:54:28.633: INFO: At 2022-12-22 03:51:16 +0000 UTC - event for pvc-f46kx: {disk.csi.azure.com_k8s-agentpool-16723883-vmss000001_2e940536-9647-4366-8108-f085cbb76ea4 } Provisioning: External provisioner is provisioning volume for claim "azuredisk-7726/pvc-f46kx"
Dec 22 03:54:28.633: INFO: At 2022-12-22 03:51:16 +0000 UTC - event for pvc-f46kx: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "disk.csi.azure.com" or manually created by system administrator
Dec 22 03:54:28.633: INFO: At 2022-12-22 03:51:18 +0000 UTC - event for pvc-f46kx: {disk.csi.azure.com_k8s-agentpool-16723883-vmss000001_2e940536-9647-4366-8108-f085cbb76ea4 } ProvisioningSucceeded: Successfully provisioned volume pvc-f0eb7302-e84a-46b1-a371-062ff124d19f
Dec 22 03:54:28.633: INFO: At 2022-12-22 03:51:19 +0000 UTC - event for azuredisk-volume-tester-fdmqt: {default-scheduler } Scheduled: Successfully assigned azuredisk-7726/azuredisk-volume-tester-fdmqt to k8s-agentpool-16723883-vmss000000
Dec 22 03:54:28.633: INFO: At 2022-12-22 03:51:30 +0000 UTC - event for azuredisk-volume-tester-fdmqt: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-f0eb7302-e84a-46b1-a371-062ff124d19f"
Dec 22 03:54:28.633: INFO: At 2022-12-22 03:51:37 +0000 UTC - event for azuredisk-volume-tester-fdmqt: {kubelet k8s-agentpool-16723883-vmss000000} Pulling: Pulling image "k8s.gcr.io/e2e-test-images/busybox:1.29-2"
Dec 22 03:54:28.633: INFO: At 2022-12-22 03:51:38 +0000 UTC - event for azuredisk-volume-tester-fdmqt: {kubelet k8s-agentpool-16723883-vmss000000} Pulled: Successfully pulled image "k8s.gcr.io/e2e-test-images/busybox:1.29-2" in 286.094193ms
Dec 22 03:54:28.633: INFO: At 2022-12-22 03:51:38 +0000 UTC - event for azuredisk-volume-tester-fdmqt: {kubelet k8s-agentpool-16723883-vmss000000} Created: Created container volume-tester
Dec 22 03:54:28.633: INFO: At 2022-12-22 03:51:38 +0000 UTC - event for azuredisk-volume-tester-fdmqt: {kubelet k8s-agentpool-16723883-vmss000000} Started: Started container volume-tester
Dec 22 03:54:28.633: INFO: At 2022-12-22 03:51:39 +0000 UTC - event for azuredisk-volume-tester-q7z48: {default-scheduler } Scheduled: Successfully assigned azuredisk-7726/azuredisk-volume-tester-q7z48 to k8s-agentpool-16723883-vmss000000
Dec 22 03:54:28.633: INFO: At 2022-12-22 03:51:39 +0000 UTC - event for volume-snapshot-kgkg4: {snapshot-controller } CreatingSnapshot: Waiting for a snapshot azuredisk-7726/volume-snapshot-kgkg4 to be created by the CSI driver.
Dec 22 03:54:28.633: INFO: At 2022-12-22 03:51:40 +0000 UTC - event for azuredisk-volume-tester-q7z48: {kubelet k8s-agentpool-16723883-vmss000000} Pulled: Successfully pulled image "k8s.gcr.io/e2e-test-images/busybox:1.29-2" in 267.773675ms
Dec 22 03:54:28.633: INFO: At 2022-12-22 03:51:40 +0000 UTC - event for azuredisk-volume-tester-q7z48: {kubelet k8s-agentpool-16723883-vmss000000} Started: Started container volume-tester
Dec 22 03:54:28.633: INFO: At 2022-12-22 03:51:40 +0000 UTC - event for azuredisk-volume-tester-q7z48: {kubelet k8s-agentpool-16723883-vmss000000} Created: Created container volume-tester
Dec 22 03:54:28.633: INFO: At 2022-12-22 03:51:40 +0000 UTC - event for azuredisk-volume-tester-q7z48: {kubelet k8s-agentpool-16723883-vmss000000} Pulling: Pulling image "k8s.gcr.io/e2e-test-images/busybox:1.29-2"
Dec 22 03:54:28.633: INFO: At 2022-12-22 03:51:42 +0000 UTC - event for volume-snapshot-kgkg4: {snapshot-controller } SnapshotCreated: Snapshot azuredisk-7726/volume-snapshot-kgkg4 was successfully created by the CSI driver.
Dec 22 03:54:28.633: INFO: At 2022-12-22 03:51:42 +0000 UTC - event for volume-snapshot-kgkg4: {snapshot-controller } SnapshotReady: Snapshot azuredisk-7726/volume-snapshot-kgkg4 is ready to use.
Dec 22 03:54:28.633: INFO: At 2022-12-22 03:51:57 +0000 UTC - event for pvc-2qvds: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "disk.csi.azure.com" or manually created by system administrator
Dec 22 03:54:28.633: INFO: At 2022-12-22 03:51:57 +0000 UTC - event for pvc-2qvds: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
Dec 22 03:54:28.633: INFO: At 2022-12-22 03:51:57 +0000 UTC - event for pvc-2qvds: {disk.csi.azure.com_k8s-agentpool-16723883-vmss000001_2e940536-9647-4366-8108-f085cbb76ea4 } Provisioning: External provisioner is provisioning volume for claim "azuredisk-7726/pvc-2qvds"
Dec 22 03:54:28.633: INFO: At 2022-12-22 03:51:59 +0000 UTC - event for pvc-2qvds: {disk.csi.azure.com_k8s-agentpool-16723883-vmss000001_2e940536-9647-4366-8108-f085cbb76ea4 } ProvisioningSucceeded: Successfully provisioned volume pvc-88a56666-9755-42d9-84c7-fc664c7c98a4
Dec 22 03:54:28.633: INFO: At 2022-12-22 03:52:00 +0000 UTC - event for azuredisk-volume-tester-vdjl5: {default-scheduler } Scheduled: Successfully assigned azuredisk-7726/azuredisk-volume-tester-vdjl5 to k8s-agentpool-16723883-vmss000001
Dec 22 03:54:28.633: INFO: At 2022-12-22 03:52:11 +0000 UTC - event for azuredisk-volume-tester-vdjl5: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-88a56666-9755-42d9-84c7-fc664c7c98a4"
Dec 22 03:54:28.633: INFO: At 2022-12-22 03:52:18 +0000 UTC - event for azuredisk-volume-tester-vdjl5: {kubelet k8s-agentpool-16723883-vmss000001} Created: Created container volume-tester
Dec 22 03:54:28.633: INFO: At 2022-12-22 03:52:18 +0000 UTC - event for azuredisk-volume-tester-vdjl5: {kubelet k8s-agentpool-16723883-vmss000001} Pulled: Successfully pulled image "k8s.gcr.io/e2e-test-images/busybox:1.29-2" in 209.985595ms
Dec 22 03:54:28.633: INFO: At 2022-12-22 03:52:18 +0000 UTC - event for azuredisk-volume-tester-vdjl5: {kubelet k8s-agentpool-16723883-vmss000001} Pulling: Pulling image "k8s.gcr.io/e2e-test-images/busybox:1.29-2"
Dec 22 03:54:28.633: INFO: At 2022-12-22 03:52:19 +0000 UTC - event for azuredisk-volume-tester-vdjl5: {kubelet k8s-agentpool-16723883-vmss000001} Started: Started container volume-tester
Dec 22 03:54:28.633: INFO: At 2022-12-22 03:52:20 +0000 UTC - event for azuredisk-volume-tester-vdjl5: {kubelet k8s-agentpool-16723883-vmss000001} Killing: Stopping container volume-tester
Dec 22 03:54:28.633: INFO: At 2022-12-22 03:53:01 +0000 UTC - event for azuredisk-volume-tester-q7z48: {kubelet k8s-agentpool-16723883-vmss000000} Killing: Stopping container volume-tester
Dec 22 03:54:28.691: INFO: POD NODE PHASE GRACE CONDITIONS
Dec 22 03:54:28.691: INFO:
Dec 22 03:54:28.749: INFO:
Logging node info for node k8s-agentpool-16723883-vmss000000
Dec 22 03:54:28.808: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool-16723883-vmss000000 d42de16e-6ee9-4c0f-bade-58d897284d06 10335 0 2022-12-22 03:03:56 +0000 UTC <nil> <nil> map[agentpool:agentpool beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D2s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-cg1hzjpb kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool-16723883-vmss000000 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: node.kubernetes.io/instance-type:Standard_D2s_v3 storageprofile:managed storagetier:Premium_LRS topology.disk.csi.azure.com/zone:westus2-1 topology.kubernetes.io/region:westus2 topology.kubernetes.io/zone:westus2-1] map[csi.volume.kubernetes.io/nodeid:{"disk.csi.azure.com":"k8s-agentpool-16723883-vmss000000"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubectl-label Update v1 2022-12-22 03:04:01 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/agent":{}}}} } {cloud-node-manager Update v1 2022-12-22 03:05:30 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-12-22 03:51:30 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}}} } {kubelet Update v1 2022-12-22 03:53:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:agentpool":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.azure.com/cluster":{},"f:kubernetes.azure.com/role":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:storageprofile":{},"f:storagetier":{},"f:topology.disk.csi.azure.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} }]},Spec:NodeSpec{PodCIDR:,DoNotUseExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-cg1hzjpb/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-16723883-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31025332224 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8344743936 0} {<nil>} 8149164Ki BinarySI},pods: {{30 0} {<nil>} 30 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27922798956 0} {<nil>} 27922798956 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7558311936 0} {<nil>} 7381164Ki BinarySI},pods: {{30 0} {<nil>} 30 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-22 03:53:39 +0000 UTC,LastTransitionTime:2022-12-22 03:03:55 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-22 03:53:39 +0000 UTC,LastTransitionTime:2022-12-22 03:03:55 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-22 03:53:39 +0000 UTC,LastTransitionTime:2022-12-22 03:03:55 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-22 03:53:39 +0000 UTC,LastTransitionTime:2022-12-22 03:04:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.248.0.4,},NodeAddress{Type:Hostname,Address:k8s-agentpool-16723883-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:72eb184066a84228baf409054c4da26d,SystemUUID:97a95192-be63-7842-ac63-623cf3ae5647,BootID:f9386e64-ae4b-4d76-ad2b-021ca0253d0a,KernelVersion:5.4.0-1098-azure,OSImage:Ubuntu 18.04.6 LTS,ContainerRuntimeVersion:containerd://1.5.13+azure-2,KubeletVersion:v1.21.14,KubeProxyVersion:v1.21.14,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi:v1.10.0],SizeBytes:250750921,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi:v1.9.0],SizeBytes:235451754,},ContainerImage{Names:[mcr.microsoft.com/containernetworking/azure-npm:v1.2.2_hotfix],SizeBytes:179170363,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-apiserver:v1.23.14],SizeBytes:136538062,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-apiserver:v1.24.8],SizeBytes:131112915,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-apiserver:v1.22.16],SizeBytes:129585107,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-apiserver:v1.21.14],SizeBytes:127000541,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-controller-manager:v1.23.14],SizeBytes:126347582,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver:v0.0.19],SizeBytes:125497261,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-controller-manager:v1.22.16],SizeBytes:123273027,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-controller-manager:v1.21.14],SizeBytes:121200451,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-controller-manager:v1.24.8],SizeBytes:120746307,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-proxy:v1.23.14],SizeBytes:114226743,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-proxy:v1.24.8],SizeBytes:111867957,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-proxy:v1.22.16],SizeBytes:105433799,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-proxy@sha256:7500b446c6391859ee81ee8fd5301605d6865f179cf47018fdea4b3119c421e1 mcr.microsoft.com/oss/kubernetes/kube-proxy:v1.21.14],SizeBytes:105351884,},ContainerImage{Names:[mcr.microsoft.com/oss/azure/secrets-store/provider-azure:0.0.12],SizeBytes:103333634,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-autoscaler:v1.22.1],SizeBytes:101308513,},ContainerImage{Names:[k8sprow.azurecr.io/azuredisk-csi@sha256:8ab802ccbbd4f5324b2ea0ade2d0d8f28d7a0d27a857a7838d3b3607cd0013e5 k8sprow.azurecr.io/azuredisk-csi:v1.26.0-98e4cd4a0b30261ca0e79e140eb9821f6ea689e2],SizeBytes:94761345,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.3],SizeBytes:88473085,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.5],SizeBytes:85735934,},ContainerImage{Names:[mcr.microsoft.com/oss/nvidia/k8s-device-plugin:1.0.0-beta6],SizeBytes:67349435,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/metrics-server:v0.5.2],SizeBytes:65672784,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner@sha256:3ef7d954946bd1cf9e5e3564a8d1acf8e5852616f7ae96bcbc5ced8c275483ee mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.3.0],SizeBytes:61391360,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-resizer@sha256:9ba6483d2f8aa6051cb3a50e42d638fc17a6e4699a6689f054969024b7c12944 mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.6.0],SizeBytes:58560473,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.0.0],SizeBytes:58145360,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-attacher@sha256:bc317fea7e7bbaff65130d7ac6ea7c96bc15eb1f086374b8c3359f11988ac024 mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v4.0.0],SizeBytes:57948644,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.3.0],SizeBytes:55577643,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v4.2.1],SizeBytes:55474791,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.3.0],SizeBytes:55106610,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-scheduler:v1.23.14],SizeBytes:54839613,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-scheduler:v1.22.16],SizeBytes:53935938,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller:v4.2.1],SizeBytes:52838996,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.23.11],SizeBytes:52722904,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.1.14],SizeBytes:52391108,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-scheduler:v1.24.8],SizeBytes:52339010,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.23.11],SizeBytes:52153533,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-scheduler:v1.21.14],SizeBytes:52130112,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.24.0],SizeBytes:50416855,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:2dfff4dc10f6d25b0a34c01bf4d166540cbf1ff2bccd52635350039138eb23c4 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.0.18],SizeBytes:50121924,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.24.0],SizeBytes:49847492,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.1.14],SizeBytes:49200344,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.0.18],SizeBytes:47197400,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/coredns:1.8.6],SizeBytes:46945346,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/mic:1.6.1],SizeBytes:42033903,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler@sha256:c849d75d61943ce7f51b4c049f1a79d19a08253966c8f49c4cfb6414cc33db8b mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler:1.8.5],SizeBytes:41925445,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/nmi:1.6.1],SizeBytes:38667001,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/ip-masq-agent@sha256:c05561241a4a87f5526ab093746558f43ee7754a4a5911a6264819e252515ea0 mcr.microsoft.com/oss/kubernetes/ip-masq-agent:v2.8.0],SizeBytes:36122811,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.16],SizeBytes:29204807,},ContainerImage{Names:[docker.io/library/registry:2.7.1],SizeBytes:26796101,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Dec 22 03:54:28.809: INFO:
Logging kubelet events for node k8s-agentpool-16723883-vmss000000
Dec 22 03:54:28.871: INFO:
Logging pods the kubelet thinks is on node k8s-agentpool-16723883-vmss000000
Dec 22 03:54:28.992: INFO: csi-azuredisk-controller-5fc74d8cf7-fzjpk started at 2022-12-22 03:19:30 +0000 UTC (0+6 container statuses recorded)
Dec 22 03:54:28.992: INFO: Container azuredisk ready: true, restart count 0
Dec 22 03:54:28.992: INFO: Container csi-attacher ready: true, restart count 0
Dec 22 03:54:28.992: INFO: Container csi-provisioner ready: true, restart count 0
Dec 22 03:54:28.992: INFO: Container csi-resizer ready: true, restart count 0
Dec 22 03:54:28.992: INFO: Container csi-snapshotter ready: true, restart count 0
Dec 22 03:54:28.992: INFO: Container liveness-probe ready: true, restart count 0
Dec 22 03:54:28.992: INFO: coredns-autoscaler-66d6fbf7c7-dgzpv started at 2022-12-22 03:04:52 +0000 UTC (0+1 container statuses recorded)
Dec 22 03:54:28.992: INFO: Container autoscaler ready: true, restart count 0
Dec 22 03:54:28.992: INFO: azure-ip-masq-agent-4rdfd started at 2022-12-22 03:04:54 +0000 UTC (0+1 container statuses recorded)
Dec 22 03:54:28.992: INFO: Container azure-ip-masq-agent ready: true, restart count 0
Dec 22 03:54:28.992: INFO: cloud-node-manager-bm799 started at 2022-12-22 03:04:54 +0000 UTC (0+1 container statuses recorded)
Dec 22 03:54:28.992: INFO: Container cloud-node-manager ready: true, restart count 0
Dec 22 03:54:28.992: INFO: kube-proxy-59j8x started at 2022-12-22 03:04:54 +0000 UTC (0+1 container statuses recorded)
Dec 22 03:54:28.992: INFO: Container kube-proxy ready: true, restart count 0
Dec 22 03:54:28.992: INFO: csi-azuredisk-node-8ld5l started at 2022-12-22 03:19:30 +0000 UTC (0+3 container statuses recorded)
Dec 22 03:54:28.992: INFO: Container azuredisk ready: true, restart count 0
Dec 22 03:54:28.992: INFO: Container liveness-probe ready: true, restart count 0
Dec 22 03:54:28.992: INFO: Container node-driver-registrar ready: true, restart count 0
Dec 22 03:54:28.992: INFO: csi-snapshot-controller-7bdf459f6f-6c9cx started at 2022-12-22 03:19:30 +0000 UTC (0+1 container statuses recorded)
Dec 22 03:54:28.992: INFO: Container csi-snapshot-controller ready: true, restart count 0
Dec 22 03:54:29.232: INFO:
Latency metrics for node k8s-agentpool-16723883-vmss000000
Dec 22 03:54:29.232: INFO:
Logging node info for node k8s-agentpool-16723883-vmss000001
Dec 22 03:54:29.291: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool-16723883-vmss000001 661a3904-2997-46aa-837f-67e111f00456 10111 0 2022-12-22 03:03:56 +0000 UTC <nil> <nil> map[agentpool:agentpool beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D2s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-cg1hzjpb kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool-16723883-vmss000001 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: node.kubernetes.io/instance-type:Standard_D2s_v3 storageprofile:managed storagetier:Premium_LRS topology.disk.csi.azure.com/zone:westus2-2 topology.kubernetes.io/region:westus2 topology.kubernetes.io/zone:westus2-2] map[csi.volume.kubernetes.io/nodeid:{"disk.csi.azure.com":"k8s-agentpool-16723883-vmss000001"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubectl-label Update v1 2022-12-22 03:04:01 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/agent":{}}}} } {cloud-node-manager Update v1 2022-12-22 03:05:30 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-12-22 03:52:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}}} } {kubelet Update v1 2022-12-22 03:52:29 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:agentpool":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.azure.com/cluster":{},"f:kubernetes.azure.com/role":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:storageprofile":{},"f:storagetier":{},"f:topology.disk.csi.azure.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} }]},Spec:NodeSpec{PodCIDR:,DoNotUseExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-cg1hzjpb/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-16723883-vmss/virtualMachines/1,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31025332224 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8344743936 0} {<nil>} 8149164Ki BinarySI},pods: {{30 0} {<nil>} 30 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27922798956 0} {<nil>} 27922798956 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7558311936 0} {<nil>} 7381164Ki BinarySI},pods: {{30 0} {<nil>} 30 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-22 03:52:29 +0000 UTC,LastTransitionTime:2022-12-22 03:03:50 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-22 03:52:29 +0000 UTC,LastTransitionTime:2022-12-22 03:03:50 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-22 03:52:29 +0000 UTC,LastTransitionTime:2022-12-22 03:03:50 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-22 03:52:29 +0000 UTC,LastTransitionTime:2022-12-22 03:03:56 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.248.0.32,},NodeAddress{Type:Hostname,Address:k8s-agentpool-16723883-vmss000001,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:8c4b1088fd0d46ffa4479ee4ba26bae2,SystemUUID:4fc18bce-a7e6-2641-b040-361db8246251,BootID:f13a3a54-144c-4d25-b2ec-29bd6dc8454f,KernelVersion:5.4.0-1098-azure,OSImage:Ubuntu 18.04.6 LTS,ContainerRuntimeVersion:containerd://1.5.13+azure-2,KubeletVersion:v1.21.14,KubeProxyVersion:v1.21.14,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi:v1.10.0],SizeBytes:250750921,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi:v1.9.0],SizeBytes:235451754,},ContainerImage{Names:[mcr.microsoft.com/containernetworking/azure-npm:v1.2.2_hotfix],SizeBytes:179170363,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-apiserver:v1.23.14],SizeBytes:136538062,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-apiserver:v1.24.8],SizeBytes:131112915,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-apiserver:v1.22.16],SizeBytes:129585107,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-apiserver:v1.21.14],SizeBytes:127000541,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-controller-manager:v1.23.14],SizeBytes:126347582,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver:v0.0.19],SizeBytes:125497261,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-controller-manager:v1.22.16],SizeBytes:123273027,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-controller-manager:v1.21.14],SizeBytes:121200451,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-controller-manager:v1.24.8],SizeBytes:120746307,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-proxy:v1.23.14],SizeBytes:114226743,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-proxy:v1.24.8],SizeBytes:111867957,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-proxy:v1.22.16],SizeBytes:105433799,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-proxy@sha256:7500b446c6391859ee81ee8fd5301605d6865f179cf47018fdea4b3119c421e1 mcr.microsoft.com/oss/kubernetes/kube-proxy:v1.21.14],SizeBytes:105351884,},ContainerImage{Names:[mcr.microsoft.com/oss/azure/secrets-store/provider-azure:0.0.12],SizeBytes:103333634,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-autoscaler:v1.22.1],SizeBytes:101308513,},ContainerImage{Names:[k8sprow.azurecr.io/azuredisk-csi@sha256:8ab802ccbbd4f5324b2ea0ade2d0d8f28d7a0d27a857a7838d3b3607cd0013e5 k8sprow.azurecr.io/azuredisk-csi:v1.26.0-98e4cd4a0b30261ca0e79e140eb9821f6ea689e2],SizeBytes:94761345,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.3],SizeBytes:88473085,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.5],SizeBytes:85735934,},ContainerImage{Names:[mcr.microsoft.com/oss/nvidia/k8s-device-plugin:1.0.0-beta6],SizeBytes:67349435,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/metrics-server@sha256:1ef9d57ce41ffcc328b92494c3bfafe401e0b9a1694a295301a1385337d52815 mcr.microsoft.com/oss/kubernetes/metrics-server:v0.5.2],SizeBytes:65672784,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner@sha256:3ef7d954946bd1cf9e5e3564a8d1acf8e5852616f7ae96bcbc5ced8c275483ee mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.3.0],SizeBytes:61391360,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-resizer@sha256:9ba6483d2f8aa6051cb3a50e42d638fc17a6e4699a6689f054969024b7c12944 mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.6.0],SizeBytes:58560473,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.0.0],SizeBytes:58145360,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-attacher@sha256:bc317fea7e7bbaff65130d7ac6ea7c96bc15eb1f086374b8c3359f11988ac024 mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v4.0.0],SizeBytes:57948644,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.3.0],SizeBytes:55577643,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v4.2.1],SizeBytes:55474791,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.3.0],SizeBytes:55106610,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-scheduler:v1.23.14],SizeBytes:54839613,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-scheduler:v1.22.16],SizeBytes:53935938,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller:v4.2.1],SizeBytes:52838996,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.23.11],SizeBytes:52722904,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.1.14],SizeBytes:52391108,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-scheduler:v1.24.8],SizeBytes:52339010,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.23.11],SizeBytes:52153533,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-scheduler:v1.21.14],SizeBytes:52130112,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.24.0],SizeBytes:50416855,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:2dfff4dc10f6d25b0a34c01bf4d166540cbf1ff2bccd52635350039138eb23c4 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.0.18],SizeBytes:50121924,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.24.0],SizeBytes:49847492,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.1.14],SizeBytes:49200344,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.0.18],SizeBytes:47197400,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/coredns@sha256:f873bf7f0928461efe10697fa76cf0ad7a1ae3041c5b57b50dd3d0b72d273f8c mcr.microsoft.com/oss/kubernetes/coredns:1.8.6],SizeBytes:46945346,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/mic:1.6.1],SizeBytes:42033903,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler:1.8.5],SizeBytes:41925445,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/nmi:1.6.1],SizeBytes:38667001,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/ip-masq-agent@sha256:c05561241a4a87f5526ab093746558f43ee7754a4a5911a6264819e252515ea0 mcr.microsoft.com/oss/kubernetes/ip-masq-agent:v2.8.0],SizeBytes:36122811,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.16],SizeBytes:29204807,},ContainerImage{Names:[docker.io/library/registry:2.7.1],SizeBytes:26796101,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Dec 22 03:54:29.293: INFO:
Logging kubelet events for node k8s-agentpool-16723883-vmss000001
Dec 22 03:54:29.355: INFO:
Logging pods the kubelet thinks is on node k8s-agentpool-16723883-vmss000001
Dec 22 03:54:29.447: INFO: csi-snapshot-controller-7bdf459f6f-ctpd7 started at 2022-12-22 03:19:30 +0000 UTC (0+1 container statuses recorded)
Dec 22 03:54:29.447: INFO: Container csi-snapshot-controller ready: true, restart count 0
Dec 22 03:54:29.447: INFO: coredns-745f545b98-mps2q started at 2022-12-22 03:04:52 +0000 UTC (0+1 container statuses recorded)
Dec 22 03:54:29.447: INFO: Container coredns ready: true, restart count 0
Dec 22 03:54:29.447: INFO: kube-proxy-2fwk6 started at 2022-12-22 03:04:54 +0000 UTC (0+1 container statuses recorded)
Dec 22 03:54:29.447: INFO: Container kube-proxy ready: true, restart count 0
Dec 22 03:54:29.447: INFO: metrics-server-7bf8c745cc-zrlnr started at 2022-12-22 03:05:30 +0000 UTC (0+1 container statuses recorded)
Dec 22 03:54:29.447: INFO: Container metrics-server ready: true, restart count 0
Dec 22 03:54:29.447: INFO: csi-azuredisk-node-hkpll started at 2022-12-22 03:19:30 +0000 UTC (0+3 container statuses recorded)
Dec 22 03:54:29.447: INFO: Container azuredisk ready: true, restart count 0
Dec 22 03:54:29.447: INFO: Container liveness-probe ready: true, restart count 0
Dec 22 03:54:29.447: INFO: Container node-driver-registrar ready: true, restart count 0
Dec 22 03:54:29.447: INFO: azure-ip-masq-agent-pfvvb started at 2022-12-22 03:04:54 +0000 UTC (0+1 container statuses recorded)
Dec 22 03:54:29.447: INFO: Container azure-ip-masq-agent ready: true, restart count 0
Dec 22 03:54:29.447: INFO: cloud-node-manager-4dbkt started at 2022-12-22 03:04:54 +0000 UTC (0+1 container statuses recorded)
Dec 22 03:54:29.447: INFO: Container cloud-node-manager ready: true, restart count 0
Dec 22 03:54:29.447: INFO: csi-azuredisk-controller-5fc74d8cf7-pp79d started at 2022-12-22 03:19:30 +0000 UTC (0+6 container statuses recorded)
Dec 22 03:54:29.447: INFO: Container azuredisk ready: true, restart count 0
Dec 22 03:54:29.447: INFO: Container csi-attacher ready: true, restart count 0
Dec 22 03:54:29.447: INFO: Container csi-provisioner ready: true, restart count 0
Dec 22 03:54:29.447: INFO: Container csi-resizer ready: true, restart count 0
Dec 22 03:54:29.447: INFO: Container csi-snapshotter ready: true, restart count 0
Dec 22 03:54:29.447: INFO: Container liveness-probe ready: true, restart count 0
Dec 22 03:54:29.738: INFO:
Latency metrics for node k8s-agentpool-16723883-vmss000001
Dec 22 03:54:29.739: INFO:
Logging node info for node k8s-master-16723883-vmss000000
Dec 22 03:54:29.797: INFO: Node Info: &Node{ObjectMeta:{k8s-master-16723883-vmss000000 52f5c93a-c787-4c9b-8603-7c6a230f0c19 9569 0 2022-12-22 03:03:56 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D2s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-cg1hzjpb kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-16723883-vmss000000 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master: node.kubernetes.io/exclude-disruption:true node.kubernetes.io/exclude-from-external-load-balancers:true node.kubernetes.io/instance-type:Standard_D2s_v3 topology.disk.csi.azure.com/zone:westus2-1 topology.kubernetes.io/region:westus2 topology.kubernetes.io/zone:westus2-1] map[csi.volume.kubernetes.io/nodeid:{"disk.csi.azure.com":"k8s-master-16723883-vmss000000"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubectl-label Update v1 2022-12-22 03:04:01 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/master":{}}}} } {kube-controller-manager Update v1 2022-12-22 03:04:06 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}}} } {cloud-node-manager Update v1 2022-12-22 03:05:32 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{},"f:taints":{}}} } {kubelet Update v1 2022-12-22 03:19:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.azure.com/cluster":{},"f:kubernetes.azure.com/role":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/exclude-disruption":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{},"f:topology.disk.csi.azure.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} }]},Spec:NodeSpec{PodCIDR:,DoNotUseExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-cg1hzjpb/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-16723883-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31025332224 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8344743936 0} {<nil>} 8149164Ki BinarySI},pods: {{30 0} {<nil>} 30 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27922798956 0} {<nil>} 27922798956 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7558311936 0} {<nil>} 7381164Ki BinarySI},pods: {{30 0} {<nil>} 30 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-22 03:50:09 +0000 UTC,LastTransitionTime:2022-12-22 03:03:48 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-22 03:50:09 +0000 UTC,LastTransitionTime:2022-12-22 03:03:48 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-22 03:50:09 +0000 UTC,LastTransitionTime:2022-12-22 03:03:48 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-22 03:50:09 +0000 UTC,LastTransitionTime:2022-12-22 03:04:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.4,},NodeAddress{Type:Hostname,Address:k8s-master-16723883-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:b9df84b6779d4a9a95c62e72482d0278,SystemUUID:d366a330-79de-b24c-bbe7-e6e57ba63a06,BootID:02759e11-260c-4a6d-8f31-7493efccc3b2,KernelVersion:5.4.0-1098-azure,OSImage:Ubuntu 18.04.6 LTS,ContainerRuntimeVersion:containerd://1.5.13+azure-2,KubeletVersion:v1.21.14,KubeProxyVersion:v1.21.14,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi:v1.10.0],SizeBytes:250750921,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi:v1.9.0],SizeBytes:235451754,},ContainerImage{Names:[mcr.microsoft.com/containernetworking/azure-npm:v1.2.2_hotfix],SizeBytes:179170363,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-apiserver:v1.23.14],SizeBytes:136538062,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-apiserver:v1.24.8],SizeBytes:131112915,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-apiserver:v1.22.16],SizeBytes:129585107,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-apiserver:v1.21.14],SizeBytes:127000541,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-controller-manager:v1.23.14],SizeBytes:126347582,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver:v0.0.19],SizeBytes:125497261,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-controller-manager:v1.22.16],SizeBytes:123273027,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-controller-manager:v1.21.14],SizeBytes:121200451,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-controller-manager:v1.24.8],SizeBytes:120746307,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-proxy:v1.23.14],SizeBytes:114226743,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-proxy:v1.24.8],SizeBytes:111867957,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-proxy:v1.22.16],SizeBytes:105433799,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-proxy@sha256:7500b446c6391859ee81ee8fd5301605d6865f179cf47018fdea4b3119c421e1 mcr.microsoft.com/oss/kubernetes/kube-proxy:v1.21.14],SizeBytes:105351884,},ContainerImage{Names:[mcr.microsoft.com/oss/azure/secrets-store/provider-azure:0.0.12],SizeBytes:103333634,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-autoscaler:v1.22.1],SizeBytes:101308513,},ContainerImage{Names:[k8sprow.azurecr.io/azuredisk-csi@sha256:8ab802ccbbd4f5324b2ea0ade2d0d8f28d7a0d27a857a7838d3b3607cd0013e5 k8sprow.azurecr.io/azuredisk-csi:v1.26.0-98e4cd4a0b30261ca0e79e140eb9821f6ea689e2],SizeBytes:94761345,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.3],SizeBytes:88473085,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.5],SizeBytes:85735934,},ContainerImage{Names:[mcr.microsoft.com/oss/nvidia/k8s-device-plugin:1.0.0-beta6],SizeBytes:67349435,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/metrics-server:v0.5.2],SizeBytes:65672784,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.0.0],SizeBytes:58145360,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.3.0],SizeBytes:55577643,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v4.2.1],SizeBytes:55474791,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.3.0],SizeBytes:55106610,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-scheduler:v1.23.14],SizeBytes:54839613,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-scheduler:v1.22.16],SizeBytes:53935938,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller:v4.2.1],SizeBytes:52838996,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.23.11],SizeBytes:52722904,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.1.14],SizeBytes:52391108,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-scheduler:v1.24.8],SizeBytes:52339010,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.23.11],SizeBytes:52153533,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-scheduler:v1.21.14],SizeBytes:52130112,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.24.0],SizeBytes:50416855,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:2dfff4dc10f6d25b0a34c01bf4d166540cbf1ff2bccd52635350039138eb23c4 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.0.18],SizeBytes:50121924,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.24.0],SizeBytes:49847492,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.1.14],SizeBytes:49200344,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.0.18],SizeBytes:47197400,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/coredns:1.8.6],SizeBytes:46945346,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/mic:1.6.1],SizeBytes:42033903,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler:1.8.5],SizeBytes:41925445,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/nmi:1.6.1],SizeBytes:38667001,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/ip-masq-agent@sha256:c05561241a4a87f5526ab093746558f43ee7754a4a5911a6264819e252515ea0 mcr.microsoft.com/oss/kubernetes/ip-masq-agent:v2.8.0],SizeBytes:36122811,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.16],SizeBytes:29204807,},ContainerImage{Names:[docker.io/library/registry:2.7.1],SizeBytes:26796101,},ContainerImage{Names:[mcr.microsoft.com/k8s/kms/keyvault:v0.0.10],SizeBytes:23356928,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar:v2.4.0],SizeBytes:20892692,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.5.0],SizeBytes:18918879,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Dec 22 03:54:29.797: INFO:
Logging kubelet events for node k8s-master-16723883-vmss000000
Dec 22 03:54:29.860: INFO:
Logging pods the kubelet thinks is on node k8s-master-16723883-vmss000000
Dec 22 03:54:29.952: INFO: cloud-controller-manager-k8s-master-16723883-vmss000000 started at 2022-12-22 03:03:36 +0000 UTC (0+1 container statuses recorded)
Dec 22 03:54:29.952: INFO: Container cloud-controller-manager ready: true, restart count 2
Dec 22 03:54:29.952: INFO: kube-scheduler-k8s-master-16723883-vmss000000 started at 2022-12-22 03:03:36 +0000 UTC (0+1 container statuses recorded)
Dec 22 03:54:29.952: INFO: Container kube-scheduler ready: true, restart count 0
Dec 22 03:54:29.952: INFO: kube-addon-manager-k8s-master-16723883-vmss000000 started at 2022-12-22 03:04:49 +0000 UTC (0+1 container statuses recorded)
Dec 22 03:54:29.952: INFO: Container kube-addon-manager ready: true, restart count 0
Dec 22 03:54:29.952: INFO: kube-apiserver-k8s-master-16723883-vmss000000 started at 2022-12-22 03:03:36 +0000 UTC (0+1 container statuses recorded)
Dec 22 03:54:29.952: INFO: Container kube-apiserver ready: true, restart count 0
Dec 22 03:54:29.952: INFO: kube-controller-manager-k8s-master-16723883-vmss000000 started at 2022-12-22 03:03:36 +0000 UTC (0+1 container statuses recorded)
Dec 22 03:54:29.952: INFO: Container kube-controller-manager ready: true, restart count 0
Dec 22 03:54:29.952: INFO: azure-ip-masq-agent-2lkp2 started at 2022-12-22 03:04:54 +0000 UTC (0+1 container statuses recorded)
Dec 22 03:54:29.952: INFO: Container azure-ip-masq-agent ready: true, restart count 0
Dec 22 03:54:29.952: INFO: cloud-node-manager-58z47 started at 2022-12-22 03:04:54 +0000 UTC (0+1 container statuses recorded)
Dec 22 03:54:29.952: INFO: Container cloud-node-manager ready: true, restart count 0
Dec 22 03:54:29.952: INFO: kube-proxy-nt5tv started at 2022-12-22 03:04:54 +0000 UTC (0+1 container statuses recorded)
Dec 22 03:54:29.952: INFO: Container kube-proxy ready: true, restart count 0
Dec 22 03:54:29.952: INFO: csi-azuredisk-node-vl4cw started at 2022-12-22 03:19:30 +0000 UTC (0+3 container statuses recorded)
Dec 22 03:54:29.952: INFO: Container azuredisk ready: true, restart count 0
Dec 22 03:54:29.952: INFO: Container liveness-probe ready: true, restart count 0
Dec 22 03:54:29.952: INFO: Container node-driver-registrar ready: true, restart count 0
Dec 22 03:54:30.213: INFO:
Latency metrics for node k8s-master-16723883-vmss000000
Dec 22 03:54:30.213: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
�[1mSTEP�[0m: Destroying namespace "azuredisk-7726" for this suite.
from junit_01.xml
Find azuredisk-volume-tester-vdjl5 mentions in log files | View test history on testgrid
error during make e2e-test: exit status 2
from junit_runner.xml
Filter through log files | View test history on testgrid
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [multi-az] Should create and attach a volume with basic perfProfile [enableBursting][disk.csi.azure.com] [Windows]
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [multi-az] Should test pod failover with cordoning a node using ZRS
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [multi-az] should clone a volume from an existing volume and read from it [disk.csi.azure.com]
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [multi-az] should clone a volume of larger size than the source volume and make sure the filesystem is appropriately adjusted [disk.csi.azure.com]
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [multi-az] should create a deployment object, write and read to it, delete the pod and write and read to it again [kubernetes.io/azure-disk] [disk.csi.azure.com] [Windows]
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [multi-az] should create a pod with multiple volumes [kubernetes.io/azure-disk] [disk.csi.azure.com] [Windows]
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [multi-az] should create a pod with volume mount subpath [disk.csi.azure.com] [Windows]
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [multi-az] should create a pod, write and read to it, take a volume snapshot, and create another pod from the snapshot [disk.csi.azure.com]
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [multi-az] should create a raw block volume and a filesystem volume on demand and bind to the same pod [kubernetes.io/azure-disk] [disk.csi.azure.com]
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [multi-az] should create a raw block volume on demand [kubernetes.io/azure-disk] [disk.csi.azure.com]
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [multi-az] should create a statefulset object, write and read to it, delete the pod and write and read to it again [kubernetes.io/azure-disk] [disk.csi.azure.com] [Windows]
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [multi-az] should create a volume azuredisk with tag [disk.csi.azure.com] [Windows]
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [multi-az] should create a volume in separate resource group and bind it to a pod [kubernetes.io/azure-disk] [disk.csi.azure.com]
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [multi-az] should create a volume on demand and mount it as readOnly in a pod [kubernetes.io/azure-disk] [disk.csi.azure.com] [Windows]
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [multi-az] should create a volume on demand and resize it [disk.csi.azure.com] [Windows]
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [multi-az] should create a volume on demand with mount options [kubernetes.io/azure-disk] [disk.csi.azure.com] [Windows]
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [multi-az] should create multiple PV objects, bind to PVCs and attach all to a single pod [kubernetes.io/azure-disk] [disk.csi.azure.com] [Windows]
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [multi-az] should create multiple PV objects, bind to PVCs and attach all to different pods on the same node [kubernetes.io/azure-disk] [disk.csi.azure.com] [Windows]
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [multi-az] should create multiple volumes, each in separate resource groups and attach them to a single pod [kubernetes.io/azure-disk] [disk.csi.azure.com]
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [multi-az] should delete PV with reclaimPolicy "Delete" [kubernetes.io/azure-disk] [disk.csi.azure.com] [Windows]
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [multi-az] should detach disk after pod deleted [disk.csi.azure.com] [Windows]
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [multi-az] should receive FailedMount event with invalid mount options [kubernetes.io/azure-disk] [disk.csi.azure.com]
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [multi-az] should retain PV with reclaimPolicy "Retain" [disk.csi.azure.com]
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [multi-az] should succeed when attaching a shared block volume to multiple pods [disk.csi.azure.com][shared disk]
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [multi-az] should succeed with advanced perfProfile [disk.csi.azure.com] [Windows]
kubetest Check APIReachability
kubetest Deferred TearDown
kubetest DumpClusterLogs
kubetest GetDeployer
kubetest IsUp
kubetest Prepare
kubetest TearDown
kubetest TearDown Previous
kubetest Timeout
kubetest Up
kubetest kubectl version
kubetest list nodes
kubetest test setup
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [multi-az] Should test pod failover with cordoning a node
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [multi-az] should create a block volume on demand and dynamically resize it without detaching [disk.csi.azure.com]
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [multi-az] should create a volume on demand and dynamically resize it without detaching [disk.csi.azure.com]
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [single-az] Should create and attach a volume with basic perfProfile [enableBursting][disk.csi.azure.com] [Windows]
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [single-az] Should test pod failover with cordoning a node
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [single-az] Should test pod failover with cordoning a node using ZRS
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [single-az] should clone a volume from an existing volume and read from it [disk.csi.azure.com]
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [single-az] should clone a volume of larger size than the source volume and make sure the filesystem is appropriately adjusted [disk.csi.azure.com]
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [single-az] should create a block volume on demand and dynamically resize it without detaching [disk.csi.azure.com]
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [single-az] should create a deployment object, write and read to it, delete the pod and write and read to it again [kubernetes.io/azure-disk] [disk.csi.azure.com] [Windows]
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [single-az] should create a pod with multiple volumes [kubernetes.io/azure-disk] [disk.csi.azure.com] [Windows]
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [single-az] should create a pod with volume mount subpath [disk.csi.azure.com] [Windows]
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [single-az] should create a pod, write and read to it, take a volume snapshot, and create another pod from the snapshot [disk.csi.azure.com]
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [single-az] should create a pod, write to its pv, take a volume snapshot, overwrite data in original pv, create another pod from the snapshot, and read unaltered original data from original pv[disk.csi.azure.com]
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [single-az] should create a raw block volume and a filesystem volume on demand and bind to the same pod [kubernetes.io/azure-disk] [disk.csi.azure.com]
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [single-az] should create a raw block volume on demand [kubernetes.io/azure-disk] [disk.csi.azure.com]
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [single-az] should create a statefulset object, write and read to it, delete the pod and write and read to it again [kubernetes.io/azure-disk] [disk.csi.azure.com] [Windows]
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [single-az] should create a volume azuredisk with tag [disk.csi.azure.com] [Windows]
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [single-az] should create a volume in separate resource group and bind it to a pod [kubernetes.io/azure-disk] [disk.csi.azure.com]
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [single-az] should create a volume on demand and dynamically resize it without detaching [disk.csi.azure.com]
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [single-az] should create a volume on demand and mount it as readOnly in a pod [kubernetes.io/azure-disk] [disk.csi.azure.com] [Windows]
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [single-az] should create a volume on demand and resize it [disk.csi.azure.com] [Windows]
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [single-az] should create a volume on demand with mount options [kubernetes.io/azure-disk] [disk.csi.azure.com] [Windows]
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [single-az] should create multiple PV objects, bind to PVCs and attach all to a single pod [kubernetes.io/azure-disk] [disk.csi.azure.com] [Windows]
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [single-az] should create multiple PV objects, bind to PVCs and attach all to different pods on the same node [kubernetes.io/azure-disk] [disk.csi.azure.com] [Windows]
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [single-az] should create multiple volumes, each in separate resource groups and attach them to a single pod [kubernetes.io/azure-disk] [disk.csi.azure.com]
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [single-az] should delete PV with reclaimPolicy "Delete" [kubernetes.io/azure-disk] [disk.csi.azure.com] [Windows]
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [single-az] should detach disk after pod deleted [disk.csi.azure.com] [Windows]
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [single-az] should receive FailedMount event with invalid mount options [kubernetes.io/azure-disk] [disk.csi.azure.com]
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [single-az] should retain PV with reclaimPolicy "Retain" [disk.csi.azure.com]
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [single-az] should succeed when attaching a shared block volume to multiple pods [disk.csi.azure.com][shared disk]
AzureDisk CSI Driver End-to-End Tests Dynamic Provisioning [single-az] should succeed with advanced perfProfile [disk.csi.azure.com] [Windows]
AzureDisk CSI Driver End-to-End Tests Pre-Provisioned [single-az] should create an inline volume by in-tree driver [kubernetes.io/azure-disk]
AzureDisk CSI Driver End-to-End Tests Pre-Provisioned [single-az] should fail when maxShares is invalid [disk.csi.azure.com][windows]
AzureDisk CSI Driver End-to-End Tests Pre-Provisioned [single-az] should succeed when attaching a shared block volume to multiple pods [disk.csi.azure.com][shared disk]
AzureDisk CSI Driver End-to-End Tests Pre-Provisioned [single-az] should succeed when creating a PremiumV2_LRS disk [disk.csi.azure.com][windows]
AzureDisk CSI Driver End-to-End Tests Pre-Provisioned [single-az] should succeed when creating a shared disk [disk.csi.azure.com][windows]
AzureDisk CSI Driver End-to-End Tests Pre-Provisioned [single-az] should succeed when reattaching a disk to a new node on DanglingAttachError [disk.csi.azure.com]
AzureDisk CSI Driver End-to-End Tests Pre-Provisioned [single-az] should use a pre-provisioned volume and mount it as readOnly in a pod [disk.csi.azure.com][windows]
AzureDisk CSI Driver End-to-End Tests Pre-Provisioned [single-az] should use a pre-provisioned volume and retain PV with reclaimPolicy "Retain" [disk.csi.azure.com][windows]