Result | FAILURE |
Tests | 13 failed / 633 succeeded |
Started | |
Elapsed | 1h37m |
Revision | |
Builder | gke-prow-ssd-pool-1a225945-qrg4 |
links | {u'resultstore': {u'url': u'https://source.cloud.google.com/results/invocations/e0fca4e8-9c34-497d-92f2-079c28d6fb61/targets/test'}} |
pod | 0d9df4c7-04bf-11ea-906a-faf80766f8b6 |
resultstore | https://source.cloud.google.com/results/invocations/e0fca4e8-9c34-497d-92f2-079c28d6fb61/targets/test |
infra-commit | 9b6e30c2f |
job-version | v1.16.3-beta.0.56+b3cbbae08ec52a-dirty |
pod | 0d9df4c7-04bf-11ea-906a-faf80766f8b6 |
repo | k8s.io/kubernetes |
repo-commit | b3cbbae08ec52a7fc73d334838e18d17e8512749 |
repos | {u'k8s.io/kubernetes': u'release-1.16', u'sigs.k8s.io/cloud-provider-azure': u'master'} |
revision | v1.16.3-beta.0.56+b3cbbae08ec52a-dirty |
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[k8s\.io\]\sPods\sshould\ssupport\spod\sreadiness\sgates\s\[NodeFeature\:PodReadinessGate\]$'
test/e2e/common/pods.go:777 Nov 11 21:08:03.019: Unexpected error: <*errors.errorString | 0xc0000a10a0>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred test/e2e/common/pods.go:811from junit_15.xml
[BeforeEach] [k8s.io] Pods test/e2e/framework/framework.go:151 �[1mSTEP�[0m: Creating a kubernetes client Nov 11 21:06:39.976: INFO: >>> kubeConfig: /workspace/aks147280421/kubeconfig/kubeconfig.westus2.json �[1mSTEP�[0m: Building a namespace api object, basename pods �[1mSTEP�[0m: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in pods-1335 �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace [BeforeEach] [k8s.io] Pods test/e2e/common/pods.go:165 [It] should support pod readiness gates [NodeFeature:PodReadinessGate] test/e2e/common/pods.go:777 �[1mSTEP�[0m: submitting the pod to kubernetes �[1mSTEP�[0m: patching pod status with condition "k8s.io/test-condition1" to true �[1mSTEP�[0m: patching pod status with condition "k8s.io/test-condition2" to true Nov 11 21:07:53.955: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Nov 11 21:07:54.958: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Nov 11 21:07:55.954: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Nov 11 21:07:56.967: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Nov 11 21:07:57.955: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Nov 11 21:07:58.956: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Nov 11 21:07:59.957: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Nov 11 21:08:00.957: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Nov 11 21:08:01.961: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Nov 11 21:08:02.954: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Nov 11 21:08:03.019: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Nov 11 21:08:03.019: FAIL: Unexpected error: <*errors.errorString | 0xc0000a10a0>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred [AfterEach] [k8s.io] Pods test/e2e/framework/framework.go:152 �[1mSTEP�[0m: Collecting events from namespace "pods-1335". �[1mSTEP�[0m: Found 5 events. Nov 11 21:08:03.097: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for pod-ready: {default-scheduler } Scheduled: Successfully assigned pods-1335/pod-ready to k8s-agentpool-35271922-vmss000001 Nov 11 21:08:03.097: INFO: At 2019-11-11 21:06:51 +0000 UTC - event for pod-ready: {kubelet k8s-agentpool-35271922-vmss000001} Pulling: Pulling image "docker.io/library/busybox:1.29" Nov 11 21:08:03.097: INFO: At 2019-11-11 21:06:54 +0000 UTC - event for pod-ready: {kubelet k8s-agentpool-35271922-vmss000001} Pulled: Successfully pulled image "docker.io/library/busybox:1.29" Nov 11 21:08:03.097: INFO: At 2019-11-11 21:07:02 +0000 UTC - event for pod-ready: {kubelet k8s-agentpool-35271922-vmss000001} Created: Created container pod-readiness-gate Nov 11 21:08:03.097: INFO: At 2019-11-11 21:07:09 +0000 UTC - event for pod-ready: {kubelet k8s-agentpool-35271922-vmss000001} Started: Started container pod-readiness-gate Nov 11 21:08:03.151: INFO: POD NODE PHASE GRACE CONDITIONS Nov 11 21:08:03.151: INFO: pod-ready k8s-agentpool-35271922-vmss000001 Running [{k8s.io/test-condition2 True 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC } {k8s.io/test-condition1 True 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC } {Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:06:40 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:06:40 +0000 UTC ReadinessGatesNotReady corresponding condition of pod readiness gate "k8s.io/test-condition1" does not exist., corresponding condition of pod readiness gate "k8s.io/test-condition2" does not exist.} {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:07:41 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:06:40 +0000 UTC }] Nov 11 21:08:03.152: INFO: Nov 11 21:08:03.315: INFO: Logging node info for node k8s-agentpool-35271922-vmss000000 Nov 11 21:08:03.370: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool-35271922-vmss000000 /api/v1/nodes/k8s-agentpool-35271922-vmss000000 04da751e-f23a-470e-b29d-e591113d7271 36068 0 2019-11-11 20:48:07 +0000 UTC <nil> <nil> map[agentpool:agentpool beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool-35271922-vmss000000 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: storageprofile:managed storagetier:Premium_LRS] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-2030":"k8s-agentpool-35271922-vmss000000","csi-hostpath-ephemeral-7436":"k8s-agentpool-35271922-vmss000000","csi-hostpath-provisioning-3597":"k8s-agentpool-35271922-vmss000000","csi-hostpath-provisioning-9334":"k8s-agentpool-35271922-vmss000000","csi-hostpath-volume-5092":"k8s-agentpool-35271922-vmss000000","csi-hostpath-volume-expand-1449":"k8s-agentpool-35271922-vmss000000","csi-mock-csi-mock-volumes-1829":"csi-mock-csi-mock-volumes-1829","csi-mock-csi-mock-volumes-2481":"csi-mock-csi-mock-volumes-2481","csi-mock-csi-mock-volumes-9721":"csi-mock-csi-mock-volumes-9721"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-35271922-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16797044736 0} {<nil>} 16403364Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16010612736 0} {<nil>} 15635364Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:08:02 +0000 UTC,LastTransitionTime:2019-11-11 20:47:58 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:08:02 +0000 UTC,LastTransitionTime:2019-11-11 20:47:58 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:08:02 +0000 UTC,LastTransitionTime:2019-11-11 20:47:58 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:08:02 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.248.0.4,},NodeAddress{Type:Hostname,Address:k8s-agentpool-35271922-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:84c06f3ee41a47aaa558c81f34711895,SystemUUID:9BB24974-637B-EE4C-A221-0F74BB743424,BootID:59f7eb0c-c09c-4eae-a14b-0710f9e14004,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[quay.io/kubernetes_incubator/nfs-provisioner@sha256:df762117e3c891f2d2ddff46ecb0776ba1f9f3c44cfd7739b0683bcd7a7954a8 quay.io/kubernetes_incubator/nfs-provisioner:v2.2.2],SizeBytes:391772778,},ContainerImage{Names:[gluster/glusterdynamic-provisioner@sha256:90067cb05a7d217651e84576935934fd9dcff8e6bcdcbaa416bbf36fcd09dbd1 gluster/glusterdynamic-provisioner:v1.0],SizeBytes:373281573,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/volume/nfs@sha256:c2ad734346f608a5f7d69cfded93c4e8094069320657bd372d12ba21dea3ea71 gcr.io/kubernetes-e2e-test-images/volume/nfs:1.0],SizeBytes:225358913,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/jessie-dnsutils@sha256:ad583e33cb284f7ef046673809b146ec4053cda19b54a85d2b180a86169715eb gcr.io/kubernetes-e2e-test-images/jessie-dnsutils:1.0],SizeBytes:195659796,},ContainerImage{Names:[httpd@sha256:addd70e4ee83f3bc9a4c1c7c41e37927ba47faf639312fc936df3afad7926f5a httpd:2.4.39-alpine],SizeBytes:126894770,},ContainerImage{Names:[httpd@sha256:eb8ccf084cf3e80eece1add239effefd171eb39adbc154d33c14260d905d4060 httpd:2.4.38-alpine],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1],SizeBytes:121711221,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727 gcr.io/kubernetes-e2e-test-images/agnhost:2.6],SizeBytes:57345321,},ContainerImage{Names:[quay.io/k8scsi/csi-provisioner@sha256:0efcb424f1dde9b9fb11a1a14f2e48ab47e1c3f08bc3a929990dcfcb1f7ab34f quay.io/k8scsi/csi-provisioner:v1.4.0-rc1],SizeBytes:54431016,},ContainerImage{Names:[quay.io/k8scsi/csi-snapshotter@sha256:e3d3e742e32d00488fdb401045b9b1d033d7ca0ab6e760f77b24750fc95e5f70 quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1],SizeBytes:51703561,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0],SizeBytes:46226754,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:0aba670b4d9d6b2e720bbf575d733156c676b693ca26501235444490300db838 quay.io/k8scsi/csi-attacher:v1.1.0],SizeBytes:42839085,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:7d46fb6eb8b890dc546029d1565d502b4a1d974d33625c6ee2bc7991b77fc1a1 quay.io/k8scsi/csi-resizer:v0.2.0],SizeBytes:42817100,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:f315c9042e56def3c05c6b04fe79ec9da6d39ddc557ca365a76cf35964ea08b6 quay.io/k8scsi/csi-resizer:v0.1.0],SizeBytes:42623056,},ContainerImage{Names:[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0],SizeBytes:42488424,},ContainerImage{Names:[quay.io/k8scsi/hostpathplugin@sha256:b4826e492fc1762fceaf9726f41575ca0a4567864d3d235da874818de18039de quay.io/k8scsi/hostpathplugin:v1.2.0-rc5],SizeBytes:28761497,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7],SizeBytes:22909487,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/echoserver@sha256:e9ba514b896cdf559eef8788b66c2c3ee55f3572df617647b4b0d8b6bf81cf19 gcr.io/kubernetes-e2e-test-images/echoserver:2.2],SizeBytes:21692741,},ContainerImage{Names:[quay.io/k8scsi/mock-driver@sha256:e0eed916b7d970bad2b7d9875f9ad16932f987f0f3d91ec5d86da68b0b5cc9d1 quay.io/k8scsi/mock-driver:v2.1.0],SizeBytes:16226335,},ContainerImage{Names:[nginx@sha256:485b610fefec7ff6c463ced9623314a04ed67e3945b9c08d7e53a47f6d108dc7 nginx:1.14-alpine],SizeBytes:16032814,},ContainerImage{Names:[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0],SizeBytes:15815995,},ContainerImage{Names:[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0],SizeBytes:14967303,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/dnsutils@sha256:2abeee84efb79c14d731966e034af33bf324d3b26ca28497555511ff094b3ddd gcr.io/kubernetes-e2e-test-images/dnsutils:1.1],SizeBytes:9349974,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nautilus@sha256:33a732d4c42a266912a5091598a0f07653c9134db4b8d571690d8afd509e0bfc gcr.io/kubernetes-e2e-test-images/nautilus:1.0],SizeBytes:4753501,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/test-webserver@sha256:7f93d6e32798ff28bc6289254d0c2867fe2c849c8e46edc50f8624734309812e gcr.io/kubernetes-e2e-test-images/test-webserver:1.0],SizeBytes:4732240,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 gcr.io/kubernetes-e2e-test-images/mounttest:1.0],SizeBytes:1563521,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29],SizeBytes:1154361,},ContainerImage{Names:[busybox@sha256:bbc3a03235220b170ba48a157dd097dd1379299370e1ed99ce976df0355d24f0 busybox:1.27],SizeBytes:1129289,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea k8s.gcr.io/pause-amd64:3.1 k8s.gcr.io/pause:3.1],SizeBytes:742472,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-provisioning-4690^f961cf89-04c6-11ea-825f-4abe8cfea59b],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-9187^4,DevicePath:,},},Config:nil,},} Nov 11 21:08:03.370: INFO: Logging kubelet events for node k8s-agentpool-35271922-vmss000000 Nov 11 21:08:03.431: INFO: Logging pods the kubelet thinks is on node k8s-agentpool-35271922-vmss000000 Nov 11 21:08:03.612: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:07:22 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:03.612: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:08:03.613: INFO: azure-ip-masq-agent-c4mb2 started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:03.613: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:08:03.613: INFO: pod-9d37ba1b-104a-4605-a060-1d3ad06112c8 started at 2019-11-11 21:07:35 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:03.613: INFO: Container test-container ready: false, restart count 0 Nov 11 21:08:03.613: INFO: rs-86vkn started at 2019-11-11 21:07:58 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:03.613: INFO: Container busybox ready: false, restart count 0 Nov 11 21:08:03.613: INFO: csi-hostpath-attacher-0 started at 2019-11-11 21:07:46 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:03.613: INFO: Container csi-attacher ready: false, restart count 0 Nov 11 21:08:03.613: INFO: ss2-0 started at 2019-11-11 21:05:49 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:03.613: INFO: Container webserver ready: true, restart count 0 Nov 11 21:08:03.613: INFO: csi-snapshotter-0 started at 2019-11-11 21:02:41 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:03.613: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 11 21:08:03.613: INFO: netserver-1 started at 2019-11-11 21:03:45 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:03.613: INFO: Container webserver ready: true, restart count 0 Nov 11 21:08:03.613: INFO: security-context-616540b3-0e9f-48f7-8af6-cfb7c75e65a6 started at 2019-11-11 21:06:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:03.613: INFO: Container write-pod ready: true, restart count 0 Nov 11 21:08:03.613: INFO: pod-secrets-76341241-4872-40e5-bd63-5cd34c00e933 started at 2019-11-11 21:07:17 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:03.613: INFO: Container secret-volume-test ready: false, restart count 0 Nov 11 21:08:03.613: INFO: keyvault-flexvolume-zzp7p started at 2019-11-11 20:48:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:03.613: INFO: Container keyvault-flexvolume ready: true, restart count 0 Nov 11 21:08:03.613: INFO: csi-hostpath-resizer-0 started at 2019-11-11 21:02:40 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:03.613: INFO: Container csi-resizer ready: true, restart count 0 Nov 11 21:08:03.613: INFO: pod-subpath-test-hostpath-fqhq started at 2019-11-11 21:08:01 +0000 UTC (1+1 container statuses recorded) Nov 11 21:08:03.613: INFO: Init container init-volume-hostpath-fqhq ready: false, restart count 0 Nov 11 21:08:03.613: INFO: Container test-container-subpath-hostpath-fqhq ready: false, restart count 0 Nov 11 21:08:03.613: INFO: rs-xgrq6 started at 2019-11-11 21:07:58 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:03.613: INFO: Container busybox ready: false, restart count 0 Nov 11 21:08:03.613: INFO: update-demo-nautilus-qr298 started at 2019-11-11 21:07:18 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:03.613: INFO: Container update-demo ready: true, restart count 0 Nov 11 21:08:03.613: INFO: pod-subpath-test-csi-hostpath-dynamicpv-ns5x started at 2019-11-11 21:07:54 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:03.613: INFO: Container test-container-subpath-csi-hostpath-dynamicpv-ns5x ready: false, restart count 0 Nov 11 21:08:03.613: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:05:18 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:03.613: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:08:03.613: INFO: kubernetes-dashboard-65966766b9-lv2x8 started at 2019-11-11 20:48:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:03.613: INFO: Container kubernetes-dashboard ready: true, restart count 0 Nov 11 21:08:03.613: INFO: rs-flh6v started at 2019-11-11 21:07:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:03.613: INFO: Container busybox ready: false, restart count 0 Nov 11 21:08:03.613: INFO: csi-hostpathplugin-0 started at 2019-11-11 21:04:26 +0000 UTC (0+3 container statuses recorded) Nov 11 21:08:03.613: INFO: Container hostpath ready: true, restart count 0 Nov 11 21:08:03.613: INFO: Container liveness-probe ready: true, restart count 0 Nov 11 21:08:03.613: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 11 21:08:03.613: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:07:21 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:03.613: INFO: Container agnhost ready: false, restart count 0 Nov 11 21:08:03.613: INFO: pvc-volume-tester-fxd2b started at 2019-11-11 21:07:54 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:03.613: INFO: Container volume-tester ready: false, restart count 0 Nov 11 21:08:03.613: INFO: csi-hostpath-resizer-0 started at 2019-11-11 21:07:52 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:03.613: INFO: Container csi-resizer ready: false, restart count 0 Nov 11 21:08:03.613: INFO: csi-hostpath-provisioner-0 started at 2019-11-11 21:02:40 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:03.613: INFO: Container csi-provisioner ready: true, restart count 0 Nov 11 21:08:03.613: INFO: pod-1 started at 2019-11-11 21:04:35 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:03.613: INFO: Container busybox ready: false, restart count 0 Nov 11 21:08:03.613: INFO: csi-mockplugin-resizer-0 started at 2019-11-11 21:06:54 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:03.613: INFO: Container csi-resizer ready: true, restart count 0 Nov 11 21:08:03.613: INFO: blobfuse-flexvol-installer-dmnhw started at 2019-11-11 20:48:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:03.613: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 Nov 11 21:08:03.613: INFO: csi-hostpath-attacher-0 started at 2019-11-11 21:02:37 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:03.613: INFO: Container csi-attacher ready: true, restart count 0 Nov 11 21:08:03.613: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:06:00 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:03.613: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:08:03.613: INFO: coredns-87f5d796-6mz2l started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:03.613: INFO: Container coredns ready: true, restart count 0 Nov 11 21:08:03.613: INFO: pod-subpath-test-local-preprovisionedpv-pqb5 started at 2019-11-11 21:06:03 +0000 UTC (2+2 container statuses recorded) Nov 11 21:08:03.613: INFO: Init container init-volume-local-preprovisionedpv-pqb5 ready: true, restart count 0 Nov 11 21:08:03.613: INFO: Init container test-init-subpath-local-preprovisionedpv-pqb5 ready: true, restart count 0 Nov 11 21:08:03.613: INFO: Container test-container-subpath-local-preprovisionedpv-pqb5 ready: false, restart count 0 Nov 11 21:08:03.613: INFO: Container test-container-volume-local-preprovisionedpv-pqb5 ready: false, restart count 0 Nov 11 21:08:03.613: INFO: pod-subpath-test-hostpathsymlink-64gk started at 2019-11-11 21:07:04 +0000 UTC (1+1 container statuses recorded) Nov 11 21:08:03.613: INFO: Init container init-volume-hostpathsymlink-64gk ready: true, restart count 0 Nov 11 21:08:03.613: INFO: Container test-container-subpath-hostpathsymlink-64gk ready: false, restart count 0 Nov 11 21:08:03.613: INFO: hostpath-symlink-prep-provisioning-4628 started at 2019-11-11 21:07:09 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:03.613: INFO: Container init-volume-provisioning-4628 ready: false, restart count 0 Nov 11 21:08:03.613: INFO: rs-l9g6r started at 2019-11-11 21:08:00 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:03.613: INFO: Container busybox ready: false, restart count 0 Nov 11 21:08:03.613: INFO: csi-hostpath-provisioner-0 started at 2019-11-11 21:07:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:03.613: INFO: Container csi-provisioner ready: false, restart count 0 Nov 11 21:08:03.613: INFO: csi-mockplugin-0 started at 2019-11-11 21:06:53 +0000 UTC (0+3 container statuses recorded) Nov 11 21:08:03.613: INFO: Container csi-provisioner ready: true, restart count 0 Nov 11 21:08:03.613: INFO: Container driver-registrar ready: true, restart count 0 Nov 11 21:08:03.613: INFO: Container mock ready: true, restart count 0 Nov 11 21:08:03.613: INFO: csi-snapshotter-0 started at 2019-11-11 21:07:53 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:03.613: INFO: Container csi-snapshotter ready: false, restart count 0 Nov 11 21:08:03.613: INFO: csi-hostpathplugin-0 started at 2019-11-11 21:07:58 +0000 UTC (0+0 container statuses recorded) Nov 11 21:08:03.613: INFO: kube-proxy-nwdpz started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:03.613: INFO: Container kube-proxy ready: true, restart count 0 W1111 21:08:03.672209 93351 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:08:05.669: INFO: Latency metrics for node k8s-agentpool-35271922-vmss000000 Nov 11 21:08:05.669: INFO: Logging node info for node k8s-agentpool-35271922-vmss000001 Nov 11 21:08:05.725: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool-35271922-vmss000001 /api/v1/nodes/k8s-agentpool-35271922-vmss000001 af5ba0b6-4f7f-4468-9d3c-b3e116ca81ca 35330 0 2019-11-11 20:48:13 +0000 UTC <nil> <nil> map[agentpool:agentpool beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool-35271922-vmss000001 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: storageprofile:managed storagetier:Premium_LRS] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-8406":"k8s-agentpool-35271922-vmss000001","csi-hostpath-provisioning-2466":"k8s-agentpool-35271922-vmss000001","csi-hostpath-provisioning-265":"k8s-agentpool-35271922-vmss000001","csi-hostpath-provisioning-310":"k8s-agentpool-35271922-vmss000001","csi-hostpath-volume-expand-3983":"k8s-agentpool-35271922-vmss000001","csi-mock-csi-mock-volumes-2781":"csi-mock-csi-mock-volumes-2781","csi-mock-csi-mock-volumes-2838":"csi-mock-csi-mock-volumes-2838","csi-mock-csi-mock-volumes-694":"csi-mock-csi-mock-volumes-694","csi-mock-csi-mock-volumes-797":"csi-mock-csi-mock-volumes-797"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-35271922-vmss/virtualMachines/1,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16797044736 0} {<nil>} 16403364Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16010612736 0} {<nil>} 15635364Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:07:27 +0000 UTC,LastTransitionTime:2019-11-11 20:48:13 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:07:27 +0000 UTC,LastTransitionTime:2019-11-11 20:48:13 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:07:27 +0000 UTC,LastTransitionTime:2019-11-11 20:48:13 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:07:27 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.248.0.5,},NodeAddress{Type:Hostname,Address:k8s-agentpool-35271922-vmss000001,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:1df8c22200b84e64b76150d232ecb929,SystemUUID:B03EE801-7A2D-7140-BE20-B2C844575A73,BootID:d32206f7-48b0-46c5-a542-39721566b456,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[quay.io/kubernetes_incubator/nfs-provisioner@sha256:df762117e3c891f2d2ddff46ecb0776ba1f9f3c44cfd7739b0683bcd7a7954a8 quay.io/kubernetes_incubator/nfs-provisioner:v2.2.2],SizeBytes:391772778,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/volume/nfs@sha256:c2ad734346f608a5f7d69cfded93c4e8094069320657bd372d12ba21dea3ea71 gcr.io/kubernetes-e2e-test-images/volume/nfs:1.0],SizeBytes:225358913,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/jessie-dnsutils@sha256:ad583e33cb284f7ef046673809b146ec4053cda19b54a85d2b180a86169715eb gcr.io/kubernetes-e2e-test-images/jessie-dnsutils:1.0],SizeBytes:195659796,},ContainerImage{Names:[httpd@sha256:addd70e4ee83f3bc9a4c1c7c41e37927ba47faf639312fc936df3afad7926f5a httpd:2.4.39-alpine],SizeBytes:126894770,},ContainerImage{Names:[httpd@sha256:eb8ccf084cf3e80eece1add239effefd171eb39adbc154d33c14260d905d4060 httpd:2.4.38-alpine],SizeBytes:123781643,},ContainerImage{Names:[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0],SizeBytes:71821984,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727 gcr.io/kubernetes-e2e-test-images/agnhost:2.6],SizeBytes:57345321,},ContainerImage{Names:[quay.io/k8scsi/csi-provisioner@sha256:0efcb424f1dde9b9fb11a1a14f2e48ab47e1c3f08bc3a929990dcfcb1f7ab34f quay.io/k8scsi/csi-provisioner:v1.4.0-rc1],SizeBytes:54431016,},ContainerImage{Names:[quay.io/k8scsi/csi-snapshotter@sha256:e3d3e742e32d00488fdb401045b9b1d033d7ca0ab6e760f77b24750fc95e5f70 quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1],SizeBytes:51703561,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0],SizeBytes:46226754,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:0aba670b4d9d6b2e720bbf575d733156c676b693ca26501235444490300db838 quay.io/k8scsi/csi-attacher:v1.1.0],SizeBytes:42839085,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:7d46fb6eb8b890dc546029d1565d502b4a1d974d33625c6ee2bc7991b77fc1a1 quay.io/k8scsi/csi-resizer:v0.2.0],SizeBytes:42817100,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:f315c9042e56def3c05c6b04fe79ec9da6d39ddc557ca365a76cf35964ea08b6 quay.io/k8scsi/csi-resizer:v0.1.0],SizeBytes:42623056,},ContainerImage{Names:[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1],SizeBytes:42541759,},ContainerImage{Names:[gcr.io/google-containers/debian-base@sha256:6966a0aedd7592c18ff2dd803c08bd85780ee19f5e3a2e7cf908a4cd837afcde gcr.io/google-containers/debian-base:0.4.1],SizeBytes:42323657,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nonroot@sha256:d4ede5c74517090b6686219059118ed178cf4620f5db8781b32f806bb1e7395b gcr.io/kubernetes-e2e-test-images/nonroot:1.0],SizeBytes:42321438,},ContainerImage{Names:[redis@sha256:50899ea1ceed33fa03232f3ac57578a424faa1742c1ac9c7a7bdb95cdf19b858 redis:5.0.5-alpine],SizeBytes:29331594,},ContainerImage{Names:[quay.io/k8scsi/hostpathplugin@sha256:b4826e492fc1762fceaf9726f41575ca0a4567864d3d235da874818de18039de quay.io/k8scsi/hostpathplugin:v1.2.0-rc5],SizeBytes:28761497,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7],SizeBytes:22909487,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/echoserver@sha256:e9ba514b896cdf559eef8788b66c2c3ee55f3572df617647b4b0d8b6bf81cf19 gcr.io/kubernetes-e2e-test-images/echoserver:2.2],SizeBytes:21692741,},ContainerImage{Names:[quay.io/k8scsi/mock-driver@sha256:e0eed916b7d970bad2b7d9875f9ad16932f987f0f3d91ec5d86da68b0b5cc9d1 quay.io/k8scsi/mock-driver:v2.1.0],SizeBytes:16226335,},ContainerImage{Names:[nginx@sha256:485b610fefec7ff6c463ced9623314a04ed67e3945b9c08d7e53a47f6d108dc7 nginx:1.14-alpine],SizeBytes:16032814,},ContainerImage{Names:[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0],SizeBytes:15815995,},ContainerImage{Names:[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0],SizeBytes:14967303,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/dnsutils@sha256:2abeee84efb79c14d731966e034af33bf324d3b26ca28497555511ff094b3ddd gcr.io/kubernetes-e2e-test-images/dnsutils:1.1],SizeBytes:9349974,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nonewprivs@sha256:10066e9039219449fe3c81f38fe01928f87914150768ab81b62a468e51fa7411 gcr.io/kubernetes-e2e-test-images/nonewprivs:1.0],SizeBytes:6757579,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nautilus@sha256:33a732d4c42a266912a5091598a0f07653c9134db4b8d571690d8afd509e0bfc gcr.io/kubernetes-e2e-test-images/nautilus:1.0],SizeBytes:4753501,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/test-webserver@sha256:7f93d6e32798ff28bc6289254d0c2867fe2c849c8e46edc50f8624734309812e gcr.io/kubernetes-e2e-test-images/test-webserver:1.0],SizeBytes:4732240,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:4206620,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 gcr.io/kubernetes-e2e-test-images/mounttest:1.0],SizeBytes:1563521,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest-user@sha256:17319ca525ee003681fccf7e8c6b1b910ff4f49b653d939ac7f9b6e7c463933d gcr.io/kubernetes-e2e-test-images/mounttest-user:1.0],SizeBytes:1450451,},ContainerImage{Names:[busybox@sha256:1303dbf110c57f3edf68d9f5a16c082ec06c4cf7604831669faf2c712260b5a0 busybox:latest],SizeBytes:1219790,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29],SizeBytes:1154361,},ContainerImage{Names:[busybox@sha256:bbc3a03235220b170ba48a157dd097dd1379299370e1ed99ce976df0355d24f0 busybox:1.27],SizeBytes:1129289,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea k8s.gcr.io/pause-amd64:3.1 k8s.gcr.io/pause:3.1],SizeBytes:742472,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-694^4],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-694^4,DevicePath:,},},Config:nil,},} Nov 11 21:08:05.726: INFO: Logging kubelet events for node k8s-agentpool-35271922-vmss000001 Nov 11 21:08:05.786: INFO: Logging pods the kubelet thinks is on node k8s-agentpool-35271922-vmss000001 Nov 11 21:08:05.962: INFO: csi-hostpath-provisioner-0 started at 2019-11-11 21:03:28 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:05.962: INFO: Container csi-provisioner ready: true, restart count 0 Nov 11 21:08:05.962: INFO: ss2-2 started at 2019-11-11 21:06:58 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:05.962: INFO: Container webserver ready: true, restart count 0 Nov 11 21:08:05.962: INFO: csi-hostpath-attacher-0 started at 2019-11-11 21:03:25 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:05.962: INFO: Container csi-attacher ready: true, restart count 0 Nov 11 21:08:05.962: INFO: busybox-scheduling-0a4d6bc3-96dc-4521-8553-1f8c895f4f3c started at 2019-11-11 21:08:00 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:05.962: INFO: Container busybox-scheduling-0a4d6bc3-96dc-4521-8553-1f8c895f4f3c ready: false, restart count 0 Nov 11 21:08:05.962: INFO: csi-snapshotter-0 started at 2019-11-11 21:03:30 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:05.962: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 11 21:08:05.962: INFO: kube-proxy-sgvk2 started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:05.962: INFO: Container kube-proxy ready: true, restart count 0 Nov 11 21:08:05.962: INFO: pod-subpath-test-local-preprovisionedpv-z7nt started at 2019-11-11 21:08:03 +0000 UTC (1+2 container statuses recorded) Nov 11 21:08:05.963: INFO: Init container test-init-subpath-local-preprovisionedpv-z7nt ready: false, restart count 0 Nov 11 21:08:05.963: INFO: Container test-container-subpath-local-preprovisionedpv-z7nt ready: false, restart count 0 Nov 11 21:08:05.963: INFO: Container test-container-volume-local-preprovisionedpv-z7nt ready: false, restart count 0 Nov 11 21:08:05.963: INFO: csi-hostpath-resizer-0 started at 2019-11-11 21:03:29 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:05.963: INFO: Container csi-resizer ready: true, restart count 0 Nov 11 21:08:05.963: INFO: pvc-volume-tester-86zrr started at 2019-11-11 21:07:18 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:05.963: INFO: Container volume-tester ready: false, restart count 0 Nov 11 21:08:05.963: INFO: csi-mockplugin-0 started at 2019-11-11 21:07:06 +0000 UTC (0+3 container statuses recorded) Nov 11 21:08:05.963: INFO: Container csi-provisioner ready: false, restart count 0 Nov 11 21:08:05.963: INFO: Container driver-registrar ready: false, restart count 0 Nov 11 21:08:05.963: INFO: Container mock ready: false, restart count 0 Nov 11 21:08:05.963: INFO: sample-crd-conversion-webhook-deployment-64d485d9bb-rqpzj started at 2019-11-11 21:07:28 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:05.963: INFO: Container sample-crd-conversion-webhook ready: false, restart count 0 Nov 11 21:08:05.963: INFO: pod-ready started at 2019-11-11 21:06:40 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:05.963: INFO: Container pod-readiness-gate ready: true, restart count 0 Nov 11 21:08:05.963: INFO: rs-t9msm started at 2019-11-11 21:07:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:05.963: INFO: Container busybox ready: false, restart count 0 Nov 11 21:08:05.963: INFO: hostexec-k8s-agentpool-35271922-vmss000001 started at 2019-11-11 21:07:15 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:05.963: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:08:05.963: INFO: rs-kcrm4 started at 2019-11-11 21:07:54 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:05.963: INFO: Container busybox ready: false, restart count 0 Nov 11 21:08:05.963: INFO: azure-ip-masq-agent-9427k started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:05.963: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:08:05.963: INFO: dns-test-18a0fc7c-f143-4aba-b036-aa00ae09a458 started at <nil> (0+0 container statuses recorded) Nov 11 21:08:05.963: INFO: tiller-deploy-7559b6b885-jsnn2 started at 2019-11-11 20:48:53 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:05.963: INFO: Container tiller ready: true, restart count 0 Nov 11 21:08:05.963: INFO: blobfuse-flexvol-installer-tvkwl started at 2019-11-11 20:48:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:05.963: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 Nov 11 21:08:05.963: INFO: rs-v7jlm started at 2019-11-11 21:07:58 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:05.963: INFO: Container busybox ready: false, restart count 0 Nov 11 21:08:05.963: INFO: metrics-server-58ff8c5ddf-thxvp started at 2019-11-11 20:48:53 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:05.963: INFO: Container metrics-server ready: true, restart count 0 Nov 11 21:08:05.963: INFO: rs-g6kzj started at 2019-11-11 21:07:56 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:05.963: INFO: Container busybox ready: false, restart count 0 Nov 11 21:08:05.963: INFO: test-webserver-5659bd3a-d0a6-41c5-9e0b-bac115ac41d4 started at 2019-11-11 21:07:42 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:05.963: INFO: Container test-webserver ready: false, restart count 0 Nov 11 21:08:05.963: INFO: rs-pkb4d started at 2019-11-11 21:07:52 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:05.963: INFO: Container busybox ready: false, restart count 0 Nov 11 21:08:05.963: INFO: rs-qx5cd started at 2019-11-11 21:07:53 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:05.963: INFO: Container busybox ready: false, restart count 0 Nov 11 21:08:05.963: INFO: csi-mockplugin-attacher-0 started at 2019-11-11 21:07:07 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:05.963: INFO: Container csi-attacher ready: true, restart count 0 Nov 11 21:08:05.963: INFO: test-container-pod started at 2019-11-11 21:05:46 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:05.963: INFO: Container webserver ready: true, restart count 0 Nov 11 21:08:05.963: INFO: csi-mockplugin-0 started at 2019-11-11 21:06:03 +0000 UTC (0+3 container statuses recorded) Nov 11 21:08:05.963: INFO: Container csi-provisioner ready: true, restart count 0 Nov 11 21:08:05.963: INFO: Container driver-registrar ready: true, restart count 0 Nov 11 21:08:05.963: INFO: Container mock ready: true, restart count 0 Nov 11 21:08:05.963: INFO: csi-hostpathplugin-0 started at 2019-11-11 21:03:26 +0000 UTC (0+3 container statuses recorded) Nov 11 21:08:05.963: INFO: Container hostpath ready: true, restart count 0 Nov 11 21:08:05.963: INFO: Container liveness-probe ready: true, restart count 0 Nov 11 21:08:05.963: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 11 21:08:05.963: INFO: csi-mockplugin-attacher-0 started at 2019-11-11 21:06:05 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:05.963: INFO: Container csi-attacher ready: true, restart count 0 Nov 11 21:08:05.963: INFO: update-demo-nautilus-qrpg5 started at 2019-11-11 21:07:18 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:05.963: INFO: Container update-demo ready: false, restart count 0 Nov 11 21:08:05.963: INFO: pod-0 started at 2019-11-11 21:06:45 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:05.963: INFO: Container busybox ready: true, restart count 0 Nov 11 21:08:05.963: INFO: ss2-1 started at 2019-11-11 21:01:47 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:05.963: INFO: Container webserver ready: false, restart count 0 Nov 11 21:08:05.963: INFO: host-test-container-pod started at 2019-11-11 21:05:48 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:05.963: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:08:05.963: INFO: pod-subpath-test-emptydir-vx7b started at 2019-11-11 21:07:45 +0000 UTC (1+1 container statuses recorded) Nov 11 21:08:05.963: INFO: Init container init-volume-emptydir-vx7b ready: false, restart count 0 Nov 11 21:08:05.963: INFO: Container test-container-subpath-emptydir-vx7b ready: false, restart count 0 Nov 11 21:08:05.963: INFO: keyvault-flexvolume-2s2wq started at 2019-11-11 20:48:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:05.963: INFO: Container keyvault-flexvolume ready: true, restart count 0 W1111 21:08:06.020636 93351 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:08:06.669: INFO: Latency metrics for node k8s-agentpool-35271922-vmss000001 Nov 11 21:08:06.669: INFO: Logging node info for node k8s-master-35271922-vmss000000 Nov 11 21:08:06.725: INFO: Node Info: &Node{ObjectMeta:{k8s-master-35271922-vmss000000 /api/v1/nodes/k8s-master-35271922-vmss000000 f042d5b7-5ee8-4528-ae46-c997fef88c1c 36025 0 2019-11-11 20:48:07 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-35271922-vmss000000 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-35271922-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284363264 0} {<nil>} 7113636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6497931264 0} {<nil>} 6345636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:08:01 +0000 UTC,LastTransitionTime:2019-11-11 20:48:04 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:08:01 +0000 UTC,LastTransitionTime:2019-11-11 20:48:04 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:08:01 +0000 UTC,LastTransitionTime:2019-11-11 20:48:04 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:08:01 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.4,},NodeAddress{Type:Hostname,Address:k8s-master-35271922-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:8843d4f5cd0544d08ae501580db6a3a7,SystemUUID:41DCC090-746D-9E44-8888-8A77F0E0C51C,BootID:42d9eb02-ab49-4c2a-8791-7c5b2ac8f2bc,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:45e5b123e6188348715630acd2f7c1469c5b5181078ce90dc6f5399be4c17be1 k8sprow.azurecr.io/azure-cloud-controller-manager:1193984044413489154-9318bf1d],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:08:06.726: INFO: Logging kubelet events for node k8s-master-35271922-vmss000000 Nov 11 21:08:06.785: INFO: Logging pods the kubelet thinks is on node k8s-master-35271922-vmss000000 Nov 11 21:08:06.876: INFO: cloud-controller-manager-k8s-master-35271922-vmss000000 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:06.876: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 11 21:08:06.876: INFO: kube-addon-manager-k8s-master-35271922-vmss000000 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:06.876: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 11 21:08:06.876: INFO: kube-apiserver-k8s-master-35271922-vmss000000 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:06.876: INFO: Container kube-apiserver ready: true, restart count 0 Nov 11 21:08:06.876: INFO: kube-controller-manager-k8s-master-35271922-vmss000000 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:06.876: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 11 21:08:06.876: INFO: kube-scheduler-k8s-master-35271922-vmss000000 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:06.876: INFO: Container kube-scheduler ready: true, restart count 0 Nov 11 21:08:06.876: INFO: azure-ip-masq-agent-k7l5b started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:06.876: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:08:06.876: INFO: kube-proxy-8hzbt started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:06.876: INFO: Container kube-proxy ready: true, restart count 0 W1111 21:08:06.933005 93351 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:08:07.075: INFO: Latency metrics for node k8s-master-35271922-vmss000000 Nov 11 21:08:07.075: INFO: Logging node info for node k8s-master-35271922-vmss000001 Nov 11 21:08:07.132: INFO: Node Info: &Node{ObjectMeta:{k8s-master-35271922-vmss000001 /api/v1/nodes/k8s-master-35271922-vmss000001 0ff06487-2e4c-4c07-a6f2-d77d1a47bbcd 36119 0 2019-11-11 20:48:39 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-35271922-vmss000001 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.6.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-35271922-vmss/virtualMachines/1,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.6.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284363264 0} {<nil>} 7113636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6497931264 0} {<nil>} 6345636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:49:01 +0000 UTC,LastTransitionTime:2019-11-11 20:49:01 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:08:03 +0000 UTC,LastTransitionTime:2019-11-11 20:48:32 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:08:03 +0000 UTC,LastTransitionTime:2019-11-11 20:48:32 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:08:03 +0000 UTC,LastTransitionTime:2019-11-11 20:48:32 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:08:03 +0000 UTC,LastTransitionTime:2019-11-11 20:48:39 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.5,},NodeAddress{Type:Hostname,Address:k8s-master-35271922-vmss000001,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:24b17940193c48e38f59d10d619fb718,SystemUUID:25FDA250-5862-9148-98C8-17FFDD874AEC,BootID:cdd7987a-6ece-464d-a177-21f53ae0b68b,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:45e5b123e6188348715630acd2f7c1469c5b5181078ce90dc6f5399be4c17be1 k8sprow.azurecr.io/azure-cloud-controller-manager:1193984044413489154-9318bf1d],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:08:07.132: INFO: Logging kubelet events for node k8s-master-35271922-vmss000001 Nov 11 21:08:07.192: INFO: Logging pods the kubelet thinks is on node k8s-master-35271922-vmss000001 Nov 11 21:08:07.272: INFO: kube-proxy-5fcm6 started at 2019-11-11 20:48:40 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:07.272: INFO: Container kube-proxy ready: true, restart count 0 Nov 11 21:08:07.272: INFO: cloud-controller-manager-k8s-master-35271922-vmss000001 started at 2019-11-11 20:48:26 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:07.272: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 11 21:08:07.272: INFO: kube-addon-manager-k8s-master-35271922-vmss000001 started at 2019-11-11 20:48:26 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:07.272: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 11 21:08:07.272: INFO: kube-apiserver-k8s-master-35271922-vmss000001 started at 2019-11-11 20:48:26 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:07.272: INFO: Container kube-apiserver ready: true, restart count 0 Nov 11 21:08:07.272: INFO: kube-controller-manager-k8s-master-35271922-vmss000001 started at 2019-11-11 20:48:26 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:07.272: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 11 21:08:07.272: INFO: kube-scheduler-k8s-master-35271922-vmss000001 started at 2019-11-11 20:48:26 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:07.272: INFO: Container kube-scheduler ready: true, restart count 0 Nov 11 21:08:07.272: INFO: azure-ip-masq-agent-bm2t5 started at 2019-11-11 20:48:40 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:07.272: INFO: Container azure-ip-masq-agent ready: true, restart count 0 W1111 21:08:07.336114 93351 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:08:07.467: INFO: Latency metrics for node k8s-master-35271922-vmss000001 Nov 11 21:08:07.467: INFO: Logging node info for node k8s-master-35271922-vmss000002 Nov 11 21:08:07.523: INFO: Node Info: &Node{ObjectMeta:{k8s-master-35271922-vmss000002 /api/v1/nodes/k8s-master-35271922-vmss000002 7fedbabf-dce6-48d1-994b-628369e7bcb5 36090 0 2019-11-11 20:48:08 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-35271922-vmss000002 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.4.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-35271922-vmss/virtualMachines/2,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284363264 0} {<nil>} 7113636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6497931264 0} {<nil>} 6345636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:08:02 +0000 UTC,LastTransitionTime:2019-11-11 20:48:03 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:08:02 +0000 UTC,LastTransitionTime:2019-11-11 20:48:03 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:08:02 +0000 UTC,LastTransitionTime:2019-11-11 20:48:03 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:08:02 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.6,},NodeAddress{Type:Hostname,Address:k8s-master-35271922-vmss000002,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:5ecd4c723b8845b18d85d341ba062204,SystemUUID:9E772AEA-D3D7-084D-98ED-1C1A511168B4,BootID:a73a4655-81c3-42e2-9d20-ef7e3cff3664,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:45e5b123e6188348715630acd2f7c1469c5b5181078ce90dc6f5399be4c17be1 k8sprow.azurecr.io/azure-cloud-controller-manager:1193984044413489154-9318bf1d],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:08:07.524: INFO: Logging kubelet events for node k8s-master-35271922-vmss000002 Nov 11 21:08:07.588: INFO: Logging pods the kubelet thinks is on node k8s-master-35271922-vmss000002 Nov 11 21:08:07.679: INFO: azure-ip-masq-agent-cmbqm started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:07.679: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:08:07.679: INFO: kube-proxy-t2sp7 started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:07.679: INFO: Container kube-proxy ready: true, restart count 0 Nov 11 21:08:07.679: INFO: cloud-controller-manager-k8s-master-35271922-vmss000002 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:07.680: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 11 21:08:07.680: INFO: kube-addon-manager-k8s-master-35271922-vmss000002 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:07.680: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 11 21:08:07.680: INFO: kube-apiserver-k8s-master-35271922-vmss000002 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:07.680: INFO: Container kube-apiserver ready: true, restart count 0 Nov 11 21:08:07.680: INFO: kube-controller-manager-k8s-master-35271922-vmss000002 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:07.680: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 11 21:08:07.680: INFO: kube-scheduler-k8s-master-35271922-vmss000002 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:07.680: INFO: Container kube-scheduler ready: true, restart count 0 W1111 21:08:07.736980 93351 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:08:07.876: INFO: Latency metrics for node k8s-master-35271922-vmss000002 Nov 11 21:08:07.876: INFO: Logging node info for node k8s-master-35271922-vmss000003 Nov 11 21:08:07.938: INFO: Node Info: &Node{ObjectMeta:{k8s-master-35271922-vmss000003 /api/v1/nodes/k8s-master-35271922-vmss000003 d4d04c51-fc7b-4cdc-a1e3-69990a198568 36141 0 2019-11-11 20:48:10 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-35271922-vmss000003 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.5.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-35271922-vmss/virtualMachines/3,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.5.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284363264 0} {<nil>} 7113636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6497931264 0} {<nil>} 6345636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:08:03 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:08:03 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:08:03 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:08:03 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.7,},NodeAddress{Type:Hostname,Address:k8s-master-35271922-vmss000003,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:53117d11ac104f35951d19e4a4448f8f,SystemUUID:048B9870-F8AE-BC47-80D8-26703FED9760,BootID:fc64edd8-d9c5-4d6f-b9e7-c902146bc724,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:45e5b123e6188348715630acd2f7c1469c5b5181078ce90dc6f5399be4c17be1 k8sprow.azurecr.io/azure-cloud-controller-manager:1193984044413489154-9318bf1d],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:08:07.938: INFO: Logging kubelet events for node k8s-master-35271922-vmss000003 Nov 11 21:08:07.998: INFO: Logging pods the kubelet thinks is on node k8s-master-35271922-vmss000003 Nov 11 21:08:08.077: INFO: cloud-controller-manager-k8s-master-35271922-vmss000003 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:08.077: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 11 21:08:08.077: INFO: kube-addon-manager-k8s-master-35271922-vmss000003 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:08.077: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 11 21:08:08.077: INFO: kube-apiserver-k8s-master-35271922-vmss000003 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:08.077: INFO: Container kube-apiserver ready: true, restart count 0 Nov 11 21:08:08.077: INFO: kube-controller-manager-k8s-master-35271922-vmss000003 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:08.077: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 11 21:08:08.077: INFO: kube-scheduler-k8s-master-35271922-vmss000003 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:08.077: INFO: Container kube-scheduler ready: true, restart count 0 Nov 11 21:08:08.077: INFO: azure-ip-masq-agent-rs668 started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:08.077: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:08:08.077: INFO: kube-proxy-tqjkf started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:08.077: INFO: Container kube-proxy ready: true, restart count 0 W1111 21:08:08.138047 93351 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:08:08.274: INFO: Latency metrics for node k8s-master-35271922-vmss000003 Nov 11 21:08:08.274: INFO: Logging node info for node k8s-master-35271922-vmss000004 Nov 11 21:08:08.331: INFO: Node Info: &Node{ObjectMeta:{k8s-master-35271922-vmss000004 /api/v1/nodes/k8s-master-35271922-vmss000004 edf29ad2-2fc8-4411-ac05-e1bf8781e340 36148 0 2019-11-11 20:48:11 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-35271922-vmss000004 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-35271922-vmss/virtualMachines/4,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284363264 0} {<nil>} 7113636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6497931264 0} {<nil>} 6345636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:08:04 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:08:04 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:08:04 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:08:04 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.8,},NodeAddress{Type:Hostname,Address:k8s-master-35271922-vmss000004,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:3ecc95ca67e14e2e87659efc6707e3b4,SystemUUID:0649BB2F-7C03-5743-A309-9BABFF3DD11A,BootID:1256b8d0-bf8e-4490-a45d-beaf41e45a09,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:45e5b123e6188348715630acd2f7c1469c5b5181078ce90dc6f5399be4c17be1 k8sprow.azurecr.io/azure-cloud-controller-manager:1193984044413489154-9318bf1d],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:08:08.331: INFO: Logging kubelet events for node k8s-master-35271922-vmss000004 Nov 11 21:08:08.390: INFO: Logging pods the kubelet thinks is on node k8s-master-35271922-vmss000004 Nov 11 21:08:08.474: INFO: kube-apiserver-k8s-master-35271922-vmss000004 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:08.474: INFO: Container kube-apiserver ready: true, restart count 0 Nov 11 21:08:08.474: INFO: kube-controller-manager-k8s-master-35271922-vmss000004 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:08.474: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 11 21:08:08.474: INFO: kube-scheduler-k8s-master-35271922-vmss000004 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:08.474: INFO: Container kube-scheduler ready: true, restart count 0 Nov 11 21:08:08.474: INFO: azure-ip-masq-agent-trzsm started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:08.474: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:08:08.474: INFO: kube-proxy-zrmqq started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:08.474: INFO: Container kube-proxy ready: true, restart count 0 Nov 11 21:08:08.474: INFO: cloud-controller-manager-k8s-master-35271922-vmss000004 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:08.474: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 11 21:08:08.474: INFO: kube-addon-manager-k8s-master-35271922-vmss000004 started at 2019-11-11 20:48:19 +0000 UTC (0+1 container statuses recorded) Nov 11 21:08:08.474: INFO: Container kube-addon-manager ready: true, restart count 0 W1111 21:08:08.532371 93351 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:08:08.664: INFO: Latency metrics for node k8s-master-35271922-vmss000004 Nov 11 21:08:08.664: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "pods-1335" for this suite. Nov 11 21:10:16.911: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered Nov 11 21:10:18.834: INFO: namespace pods-1335 deletion completed in 2m10.107572544s
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[k8s\.io\]\sPods\sshould\ssupport\spod\sreadiness\sgates\s\[NodeFeature\:PodReadinessGate\]$'
test/e2e/common/pods.go:777 Nov 11 21:11:25.917: Unexpected error: <*errors.errorString | 0xc0000a10a0>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred test/e2e/common/pods.go:811from junit_15.xml
[BeforeEach] [k8s.io] Pods test/e2e/framework/framework.go:151 �[1mSTEP�[0m: Creating a kubernetes client Nov 11 21:10:18.841: INFO: >>> kubeConfig: /workspace/aks147280421/kubeconfig/kubeconfig.westus2.json �[1mSTEP�[0m: Building a namespace api object, basename pods �[1mSTEP�[0m: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in pods-4997 �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace [BeforeEach] [k8s.io] Pods test/e2e/common/pods.go:165 [It] should support pod readiness gates [NodeFeature:PodReadinessGate] test/e2e/common/pods.go:777 �[1mSTEP�[0m: submitting the pod to kubernetes �[1mSTEP�[0m: patching pod status with condition "k8s.io/test-condition1" to true �[1mSTEP�[0m: patching pod status with condition "k8s.io/test-condition2" to true Nov 11 21:11:16.862: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Nov 11 21:11:17.864: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Nov 11 21:11:18.862: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Nov 11 21:11:19.864: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Nov 11 21:11:20.862: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Nov 11 21:11:21.861: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Nov 11 21:11:22.873: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Nov 11 21:11:23.862: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Nov 11 21:11:24.861: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Nov 11 21:11:25.862: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Nov 11 21:11:25.917: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Nov 11 21:11:25.917: FAIL: Unexpected error: <*errors.errorString | 0xc0000a10a0>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred [AfterEach] [k8s.io] Pods test/e2e/framework/framework.go:152 �[1mSTEP�[0m: Collecting events from namespace "pods-4997". �[1mSTEP�[0m: Found 5 events. Nov 11 21:11:25.974: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for pod-ready: {default-scheduler } Scheduled: Successfully assigned pods-4997/pod-ready to k8s-agentpool-35271922-vmss000001 Nov 11 21:11:25.974: INFO: At 2019-11-11 21:10:50 +0000 UTC - event for pod-ready: {kubelet k8s-agentpool-35271922-vmss000001} Pulling: Pulling image "docker.io/library/busybox:1.29" Nov 11 21:11:25.974: INFO: At 2019-11-11 21:10:51 +0000 UTC - event for pod-ready: {kubelet k8s-agentpool-35271922-vmss000001} Pulled: Successfully pulled image "docker.io/library/busybox:1.29" Nov 11 21:11:25.974: INFO: At 2019-11-11 21:10:57 +0000 UTC - event for pod-ready: {kubelet k8s-agentpool-35271922-vmss000001} Created: Created container pod-readiness-gate Nov 11 21:11:25.974: INFO: At 2019-11-11 21:11:02 +0000 UTC - event for pod-ready: {kubelet k8s-agentpool-35271922-vmss000001} Started: Started container pod-readiness-gate Nov 11 21:11:26.029: INFO: POD NODE PHASE GRACE CONDITIONS Nov 11 21:11:26.029: INFO: pod-ready k8s-agentpool-35271922-vmss000001 Running [{k8s.io/test-condition2 True 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC } {k8s.io/test-condition1 True 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC } {Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:10:19 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:10:19 +0000 UTC ReadinessGatesNotReady corresponding condition of pod readiness gate "k8s.io/test-condition1" does not exist., corresponding condition of pod readiness gate "k8s.io/test-condition2" does not exist.} {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:11:03 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:10:19 +0000 UTC }] Nov 11 21:11:26.029: INFO: Nov 11 21:11:26.192: INFO: Logging node info for node k8s-agentpool-35271922-vmss000000 Nov 11 21:11:26.248: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool-35271922-vmss000000 /api/v1/nodes/k8s-agentpool-35271922-vmss000000 04da751e-f23a-470e-b29d-e591113d7271 39237 0 2019-11-11 20:48:07 +0000 UTC <nil> <nil> map[agentpool:agentpool beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubelet_cleanup:true kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool-35271922-vmss000000 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: storageprofile:managed storagetier:Premium_LRS] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-2030":"k8s-agentpool-35271922-vmss000000","csi-hostpath-ephemeral-7436":"k8s-agentpool-35271922-vmss000000","csi-hostpath-provisioning-3597":"k8s-agentpool-35271922-vmss000000","csi-hostpath-provisioning-9334":"k8s-agentpool-35271922-vmss000000","csi-hostpath-volume-5092":"k8s-agentpool-35271922-vmss000000","csi-hostpath-volume-expand-1449":"k8s-agentpool-35271922-vmss000000","csi-mock-csi-mock-volumes-1829":"csi-mock-csi-mock-volumes-1829","csi-mock-csi-mock-volumes-2481":"csi-mock-csi-mock-volumes-2481","csi-mock-csi-mock-volumes-9721":"csi-mock-csi-mock-volumes-9721"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-35271922-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16797044736 0} {<nil>} 16403364Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16010612736 0} {<nil>} 15635364Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:11:02 +0000 UTC,LastTransitionTime:2019-11-11 20:47:58 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:11:02 +0000 UTC,LastTransitionTime:2019-11-11 20:47:58 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:11:02 +0000 UTC,LastTransitionTime:2019-11-11 20:47:58 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:11:02 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.248.0.4,},NodeAddress{Type:Hostname,Address:k8s-agentpool-35271922-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:84c06f3ee41a47aaa558c81f34711895,SystemUUID:9BB24974-637B-EE4C-A221-0F74BB743424,BootID:59f7eb0c-c09c-4eae-a14b-0710f9e14004,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[quay.io/kubernetes_incubator/nfs-provisioner@sha256:df762117e3c891f2d2ddff46ecb0776ba1f9f3c44cfd7739b0683bcd7a7954a8 quay.io/kubernetes_incubator/nfs-provisioner:v2.2.2],SizeBytes:391772778,},ContainerImage{Names:[gluster/glusterdynamic-provisioner@sha256:90067cb05a7d217651e84576935934fd9dcff8e6bcdcbaa416bbf36fcd09dbd1 gluster/glusterdynamic-provisioner:v1.0],SizeBytes:373281573,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/volume/nfs@sha256:c2ad734346f608a5f7d69cfded93c4e8094069320657bd372d12ba21dea3ea71 gcr.io/kubernetes-e2e-test-images/volume/nfs:1.0],SizeBytes:225358913,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/jessie-dnsutils@sha256:ad583e33cb284f7ef046673809b146ec4053cda19b54a85d2b180a86169715eb gcr.io/kubernetes-e2e-test-images/jessie-dnsutils:1.0],SizeBytes:195659796,},ContainerImage{Names:[httpd@sha256:addd70e4ee83f3bc9a4c1c7c41e37927ba47faf639312fc936df3afad7926f5a httpd:2.4.39-alpine],SizeBytes:126894770,},ContainerImage{Names:[httpd@sha256:eb8ccf084cf3e80eece1add239effefd171eb39adbc154d33c14260d905d4060 httpd:2.4.38-alpine],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1],SizeBytes:121711221,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727 gcr.io/kubernetes-e2e-test-images/agnhost:2.6],SizeBytes:57345321,},ContainerImage{Names:[quay.io/k8scsi/csi-provisioner@sha256:0efcb424f1dde9b9fb11a1a14f2e48ab47e1c3f08bc3a929990dcfcb1f7ab34f quay.io/k8scsi/csi-provisioner:v1.4.0-rc1],SizeBytes:54431016,},ContainerImage{Names:[quay.io/k8scsi/csi-snapshotter@sha256:e3d3e742e32d00488fdb401045b9b1d033d7ca0ab6e760f77b24750fc95e5f70 quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1],SizeBytes:51703561,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0],SizeBytes:46226754,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:0aba670b4d9d6b2e720bbf575d733156c676b693ca26501235444490300db838 quay.io/k8scsi/csi-attacher:v1.1.0],SizeBytes:42839085,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:7d46fb6eb8b890dc546029d1565d502b4a1d974d33625c6ee2bc7991b77fc1a1 quay.io/k8scsi/csi-resizer:v0.2.0],SizeBytes:42817100,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:f315c9042e56def3c05c6b04fe79ec9da6d39ddc557ca365a76cf35964ea08b6 quay.io/k8scsi/csi-resizer:v0.1.0],SizeBytes:42623056,},ContainerImage{Names:[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0],SizeBytes:42488424,},ContainerImage{Names:[quay.io/k8scsi/hostpathplugin@sha256:b4826e492fc1762fceaf9726f41575ca0a4567864d3d235da874818de18039de quay.io/k8scsi/hostpathplugin:v1.2.0-rc5],SizeBytes:28761497,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7],SizeBytes:22909487,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/echoserver@sha256:e9ba514b896cdf559eef8788b66c2c3ee55f3572df617647b4b0d8b6bf81cf19 gcr.io/kubernetes-e2e-test-images/echoserver:2.2],SizeBytes:21692741,},ContainerImage{Names:[quay.io/k8scsi/mock-driver@sha256:e0eed916b7d970bad2b7d9875f9ad16932f987f0f3d91ec5d86da68b0b5cc9d1 quay.io/k8scsi/mock-driver:v2.1.0],SizeBytes:16226335,},ContainerImage{Names:[nginx@sha256:485b610fefec7ff6c463ced9623314a04ed67e3945b9c08d7e53a47f6d108dc7 nginx:1.14-alpine],SizeBytes:16032814,},ContainerImage{Names:[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0],SizeBytes:15815995,},ContainerImage{Names:[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0],SizeBytes:14967303,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/dnsutils@sha256:2abeee84efb79c14d731966e034af33bf324d3b26ca28497555511ff094b3ddd gcr.io/kubernetes-e2e-test-images/dnsutils:1.1],SizeBytes:9349974,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nautilus@sha256:33a732d4c42a266912a5091598a0f07653c9134db4b8d571690d8afd509e0bfc gcr.io/kubernetes-e2e-test-images/nautilus:1.0],SizeBytes:4753501,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/test-webserver@sha256:7f93d6e32798ff28bc6289254d0c2867fe2c849c8e46edc50f8624734309812e gcr.io/kubernetes-e2e-test-images/test-webserver:1.0],SizeBytes:4732240,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 gcr.io/kubernetes-e2e-test-images/mounttest:1.0],SizeBytes:1563521,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29],SizeBytes:1154361,},ContainerImage{Names:[busybox@sha256:bbc3a03235220b170ba48a157dd097dd1379299370e1ed99ce976df0355d24f0 busybox:1.27],SizeBytes:1129289,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea k8s.gcr.io/pause-amd64:3.1 k8s.gcr.io/pause:3.1],SizeBytes:742472,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-provisioning-4690^f961cf89-04c6-11ea-825f-4abe8cfea59b],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-9187^4,DevicePath:,},},Config:nil,},} Nov 11 21:11:26.249: INFO: Logging kubelet events for node k8s-agentpool-35271922-vmss000000 Nov 11 21:11:26.316: INFO: Logging pods the kubelet thinks is on node k8s-agentpool-35271922-vmss000000 Nov 11 21:11:26.434: INFO: rs-flh6v started at 2019-11-11 21:07:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.434: INFO: Container busybox ready: true, restart count 0 Nov 11 21:11:26.434: INFO: csi-snapshotter-0 started at 2019-11-11 21:10:29 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.434: INFO: Container csi-snapshotter ready: false, restart count 0 Nov 11 21:11:26.434: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-sqdt4 started at 2019-11-11 21:09:54 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.434: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: true, restart count 0 Nov 11 21:11:26.434: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:07:21 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.434: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:11:26.434: INFO: condition-test-8z48j started at 2019-11-11 21:11:21 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.434: INFO: Container httpd ready: false, restart count 0 Nov 11 21:11:26.434: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-7ndbp started at 2019-11-11 21:09:58 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.434: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: true, restart count 0 Nov 11 21:11:26.434: INFO: csi-hostpathplugin-0 started at 2019-11-11 21:04:26 +0000 UTC (0+3 container statuses recorded) Nov 11 21:11:26.434: INFO: Container hostpath ready: true, restart count 0 Nov 11 21:11:26.434: INFO: Container liveness-probe ready: true, restart count 0 Nov 11 21:11:26.434: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 11 21:11:26.434: INFO: externalname-service-2mr2n started at 2019-11-11 21:10:33 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.434: INFO: Container externalname-service ready: false, restart count 0 Nov 11 21:11:26.434: INFO: pod-subpath-test-local-preprovisionedpv-qjr5 started at 2019-11-11 21:10:07 +0000 UTC (2+1 container statuses recorded) Nov 11 21:11:26.434: INFO: Init container init-volume-local-preprovisionedpv-qjr5 ready: false, restart count 0 Nov 11 21:11:26.434: INFO: Init container test-init-volume-local-preprovisionedpv-qjr5 ready: false, restart count 0 Nov 11 21:11:26.434: INFO: Container test-container-subpath-local-preprovisionedpv-qjr5 ready: false, restart count 0 Nov 11 21:11:26.434: INFO: csi-hostpath-resizer-0 started at 2019-11-11 21:07:52 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.434: INFO: Container csi-resizer ready: true, restart count 0 Nov 11 21:11:26.434: INFO: pod-subpath-test-local-preprovisionedpv-l5q4 started at 2019-11-11 21:09:20 +0000 UTC (1+1 container statuses recorded) Nov 11 21:11:26.434: INFO: Init container init-volume-local-preprovisionedpv-l5q4 ready: true, restart count 0 Nov 11 21:11:26.434: INFO: Container test-container-subpath-local-preprovisionedpv-l5q4 ready: false, restart count 0 Nov 11 21:11:26.434: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-n58zl started at 2019-11-11 21:09:53 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.434: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: true, restart count 0 Nov 11 21:11:26.434: INFO: csi-hostpath-resizer-0 started at 2019-11-11 21:10:28 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.434: INFO: Container csi-resizer ready: false, restart count 0 Nov 11 21:11:26.434: INFO: csi-hostpathplugin-0 started at 2019-11-11 21:11:21 +0000 UTC (0+0 container statuses recorded) Nov 11 21:11:26.434: INFO: blobfuse-flexvol-installer-dmnhw started at 2019-11-11 20:48:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.434: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 Nov 11 21:11:26.434: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-vdczm started at 2019-11-11 21:10:00 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.435: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: false, restart count 0 Nov 11 21:11:26.435: INFO: csi-hostpath-provisioner-0 started at 2019-11-11 21:02:40 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.435: INFO: Container csi-provisioner ready: true, restart count 0 Nov 11 21:11:26.435: INFO: csi-mockplugin-resizer-0 started at 2019-11-11 21:06:54 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.435: INFO: Container csi-resizer ready: true, restart count 0 Nov 11 21:11:26.435: INFO: coredns-87f5d796-6mz2l started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.435: INFO: Container coredns ready: true, restart count 0 Nov 11 21:11:26.435: INFO: csi-hostpath-attacher-0 started at 2019-11-11 21:02:37 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.435: INFO: Container csi-attacher ready: true, restart count 0 Nov 11 21:11:26.435: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:08:10 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.435: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:11:26.435: INFO: csi-hostpath-resizer-0 started at 2019-11-11 21:08:37 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.435: INFO: Container csi-resizer ready: true, restart count 0 Nov 11 21:11:26.435: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-695c7 started at 2019-11-11 21:09:56 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.435: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: false, restart count 0 Nov 11 21:11:26.435: INFO: csi-hostpath-provisioner-0 started at 2019-11-11 21:07:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.435: INFO: Container csi-provisioner ready: true, restart count 0 Nov 11 21:11:26.435: INFO: rs-l9g6r started at 2019-11-11 21:08:00 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.435: INFO: Container busybox ready: true, restart count 0 Nov 11 21:11:26.435: INFO: csi-hostpath-attacher-0 started at 2019-11-11 21:08:34 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.435: INFO: Container csi-attacher ready: true, restart count 0 Nov 11 21:11:26.435: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-77l5m started at 2019-11-11 21:09:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.435: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: true, restart count 0 Nov 11 21:11:26.435: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-fmq2z started at 2019-11-11 21:09:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.435: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: true, restart count 0 Nov 11 21:11:26.435: INFO: kube-proxy-nwdpz started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.435: INFO: Container kube-proxy ready: true, restart count 0 Nov 11 21:11:26.435: INFO: csi-mockplugin-0 started at 2019-11-11 21:06:53 +0000 UTC (0+3 container statuses recorded) Nov 11 21:11:26.435: INFO: Container csi-provisioner ready: false, restart count 0 Nov 11 21:11:26.435: INFO: Container driver-registrar ready: false, restart count 0 Nov 11 21:11:26.435: INFO: Container mock ready: false, restart count 0 Nov 11 21:11:26.435: INFO: csi-snapshotter-0 started at 2019-11-11 21:07:53 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.435: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 11 21:11:26.435: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-6wrg6 started at 2019-11-11 21:09:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.435: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: true, restart count 0 Nov 11 21:11:26.435: INFO: local-client started at 2019-11-11 21:10:53 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.435: INFO: Container local-client ready: false, restart count 0 Nov 11 21:11:26.435: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-h7bv7 started at 2019-11-11 21:10:01 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.435: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: false, restart count 0 Nov 11 21:11:26.435: INFO: csi-mockplugin-attacher-0 started at 2019-11-11 21:11:13 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.435: INFO: Container csi-attacher ready: false, restart count 0 Nov 11 21:11:26.435: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:10:12 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.435: INFO: Container agnhost ready: false, restart count 0 Nov 11 21:11:26.435: INFO: csi-hostpathplugin-0 started at <nil> (0+0 container statuses recorded) Nov 11 21:11:26.435: INFO: azure-ip-masq-agent-c4mb2 started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.435: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:11:26.435: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-rlsfg started at 2019-11-11 21:09:55 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.435: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: true, restart count 0 Nov 11 21:11:26.435: INFO: csi-mockplugin-resizer-0 started at 2019-11-11 21:11:14 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.435: INFO: Container csi-resizer ready: false, restart count 0 Nov 11 21:11:26.435: INFO: csi-hostpathplugin-0 started at 2019-11-11 21:11:24 +0000 UTC (0+0 container statuses recorded) Nov 11 21:11:26.435: INFO: csi-hostpath-provisioner-0 started at 2019-11-11 21:08:36 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.435: INFO: Container csi-provisioner ready: true, restart count 0 Nov 11 21:11:26.435: INFO: security-context-cf3e90cf-15ec-490f-8c8f-1225dbacd3c8 started at 2019-11-11 21:10:07 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.435: INFO: Container write-pod ready: false, restart count 0 Nov 11 21:11:26.435: INFO: rs-86vkn started at 2019-11-11 21:07:58 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.435: INFO: Container busybox ready: true, restart count 0 Nov 11 21:11:26.435: INFO: csi-hostpath-provisioner-0 started at 2019-11-11 21:10:26 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.435: INFO: Container csi-provisioner ready: false, restart count 0 Nov 11 21:11:26.435: INFO: ss2-0 started at 2019-11-11 21:10:44 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.435: INFO: Container webserver ready: false, restart count 0 Nov 11 21:11:26.435: INFO: csi-mockplugin-0 started at 2019-11-11 21:11:12 +0000 UTC (0+3 container statuses recorded) Nov 11 21:11:26.435: INFO: Container csi-provisioner ready: false, restart count 0 Nov 11 21:11:26.435: INFO: Container driver-registrar ready: false, restart count 0 Nov 11 21:11:26.435: INFO: Container mock ready: false, restart count 0 Nov 11 21:11:26.435: INFO: keyvault-flexvolume-zzp7p started at 2019-11-11 20:48:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.435: INFO: Container keyvault-flexvolume ready: true, restart count 0 Nov 11 21:11:26.435: INFO: csi-hostpath-attacher-0 started at 2019-11-11 21:07:46 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.435: INFO: Container csi-attacher ready: true, restart count 0 Nov 11 21:11:26.435: INFO: csi-snapshotter-0 started at 2019-11-11 21:02:41 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.435: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 11 21:11:26.435: INFO: security-context-cb64371f-4ea5-4c1b-b7d7-7cd22c118756 started at 2019-11-11 21:09:22 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.435: INFO: Container write-pod ready: true, restart count 0 Nov 11 21:11:26.435: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:08:24 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.435: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:11:26.435: INFO: csi-hostpath-attacher-0 started at 2019-11-11 21:10:24 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.435: INFO: Container csi-attacher ready: false, restart count 0 Nov 11 21:11:26.435: INFO: rs-cnbnm started at 2019-11-11 21:10:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.435: INFO: Container busybox ready: false, restart count 0 Nov 11 21:11:26.435: INFO: csi-hostpath-resizer-0 started at 2019-11-11 21:02:40 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.435: INFO: Container csi-resizer ready: true, restart count 0 Nov 11 21:11:26.435: INFO: rs-xgrq6 started at 2019-11-11 21:07:58 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.435: INFO: Container busybox ready: true, restart count 0 Nov 11 21:11:26.435: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:09:14 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.435: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:11:26.435: INFO: hostpath-symlink-prep-provisioning-4628 started at 2019-11-11 21:10:07 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.435: INFO: Container init-volume-provisioning-4628 ready: false, restart count 0 Nov 11 21:11:26.435: INFO: kubernetes-dashboard-65966766b9-lv2x8 started at 2019-11-11 20:48:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.435: INFO: Container kubernetes-dashboard ready: true, restart count 0 Nov 11 21:11:26.435: INFO: update-demo-nautilus-qr298 started at 2019-11-11 21:07:18 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.435: INFO: Container update-demo ready: true, restart count 0 Nov 11 21:11:26.435: INFO: pod-subpath-test-csi-hostpath-dynamicpv-ns5x started at 2019-11-11 21:07:54 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.435: INFO: Container test-container-subpath-csi-hostpath-dynamicpv-ns5x ready: false, restart count 0 Nov 11 21:11:26.435: INFO: csi-snapshotter-0 started at 2019-11-11 21:08:38 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:26.435: INFO: Container csi-snapshotter ready: true, restart count 0 W1111 21:11:26.491541 93351 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:11:27.135: INFO: Latency metrics for node k8s-agentpool-35271922-vmss000000 Nov 11 21:11:27.135: INFO: Logging node info for node k8s-agentpool-35271922-vmss000001 Nov 11 21:11:27.191: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool-35271922-vmss000001 /api/v1/nodes/k8s-agentpool-35271922-vmss000001 af5ba0b6-4f7f-4468-9d3c-b3e116ca81ca 39650 0 2019-11-11 20:48:13 +0000 UTC <nil> <nil> map[agentpool:agentpool beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 fizz:buzz foo:bar kubelet_cleanup:true kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool-35271922-vmss000001 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: storageprofile:managed storagetier:Premium_LRS] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-8406":"k8s-agentpool-35271922-vmss000001","csi-hostpath-provisioning-2466":"k8s-agentpool-35271922-vmss000001","csi-hostpath-provisioning-265":"k8s-agentpool-35271922-vmss000001","csi-hostpath-provisioning-310":"k8s-agentpool-35271922-vmss000001","csi-hostpath-volume-expand-3983":"k8s-agentpool-35271922-vmss000001","csi-mock-csi-mock-volumes-2781":"csi-mock-csi-mock-volumes-2781","csi-mock-csi-mock-volumes-2838":"csi-mock-csi-mock-volumes-2838","csi-mock-csi-mock-volumes-3306":"csi-mock-csi-mock-volumes-3306","csi-mock-csi-mock-volumes-6769":"csi-mock-csi-mock-volumes-6769","csi-mock-csi-mock-volumes-694":"csi-mock-csi-mock-volumes-694","csi-mock-csi-mock-volumes-797":"csi-mock-csi-mock-volumes-797"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-35271922-vmss/virtualMachines/1,Unschedulable:false,Taints:[]Taint{Taint{Key:foo,Value:bar,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16797044736 0} {<nil>} 16403364Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16010612736 0} {<nil>} 15635364Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:11:18 +0000 UTC,LastTransitionTime:2019-11-11 20:48:13 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:11:18 +0000 UTC,LastTransitionTime:2019-11-11 20:48:13 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:11:18 +0000 UTC,LastTransitionTime:2019-11-11 20:48:13 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:11:18 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.248.0.5,},NodeAddress{Type:Hostname,Address:k8s-agentpool-35271922-vmss000001,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:1df8c22200b84e64b76150d232ecb929,SystemUUID:B03EE801-7A2D-7140-BE20-B2C844575A73,BootID:d32206f7-48b0-46c5-a542-39721566b456,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[quay.io/kubernetes_incubator/nfs-provisioner@sha256:df762117e3c891f2d2ddff46ecb0776ba1f9f3c44cfd7739b0683bcd7a7954a8 quay.io/kubernetes_incubator/nfs-provisioner:v2.2.2],SizeBytes:391772778,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/volume/nfs@sha256:c2ad734346f608a5f7d69cfded93c4e8094069320657bd372d12ba21dea3ea71 gcr.io/kubernetes-e2e-test-images/volume/nfs:1.0],SizeBytes:225358913,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/jessie-dnsutils@sha256:ad583e33cb284f7ef046673809b146ec4053cda19b54a85d2b180a86169715eb gcr.io/kubernetes-e2e-test-images/jessie-dnsutils:1.0],SizeBytes:195659796,},ContainerImage{Names:[httpd@sha256:addd70e4ee83f3bc9a4c1c7c41e37927ba47faf639312fc936df3afad7926f5a httpd:2.4.39-alpine],SizeBytes:126894770,},ContainerImage{Names:[httpd@sha256:eb8ccf084cf3e80eece1add239effefd171eb39adbc154d33c14260d905d4060 httpd:2.4.38-alpine],SizeBytes:123781643,},ContainerImage{Names:[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0],SizeBytes:71821984,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727 gcr.io/kubernetes-e2e-test-images/agnhost:2.6],SizeBytes:57345321,},ContainerImage{Names:[quay.io/k8scsi/csi-provisioner@sha256:0efcb424f1dde9b9fb11a1a14f2e48ab47e1c3f08bc3a929990dcfcb1f7ab34f quay.io/k8scsi/csi-provisioner:v1.4.0-rc1],SizeBytes:54431016,},ContainerImage{Names:[quay.io/k8scsi/csi-snapshotter@sha256:e3d3e742e32d00488fdb401045b9b1d033d7ca0ab6e760f77b24750fc95e5f70 quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1],SizeBytes:51703561,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0],SizeBytes:46226754,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:0aba670b4d9d6b2e720bbf575d733156c676b693ca26501235444490300db838 quay.io/k8scsi/csi-attacher:v1.1.0],SizeBytes:42839085,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:7d46fb6eb8b890dc546029d1565d502b4a1d974d33625c6ee2bc7991b77fc1a1 quay.io/k8scsi/csi-resizer:v0.2.0],SizeBytes:42817100,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:f315c9042e56def3c05c6b04fe79ec9da6d39ddc557ca365a76cf35964ea08b6 quay.io/k8scsi/csi-resizer:v0.1.0],SizeBytes:42623056,},ContainerImage{Names:[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1],SizeBytes:42541759,},ContainerImage{Names:[gcr.io/google-containers/debian-base@sha256:6966a0aedd7592c18ff2dd803c08bd85780ee19f5e3a2e7cf908a4cd837afcde gcr.io/google-containers/debian-base:0.4.1],SizeBytes:42323657,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nonroot@sha256:d4ede5c74517090b6686219059118ed178cf4620f5db8781b32f806bb1e7395b gcr.io/kubernetes-e2e-test-images/nonroot:1.0],SizeBytes:42321438,},ContainerImage{Names:[redis@sha256:50899ea1ceed33fa03232f3ac57578a424faa1742c1ac9c7a7bdb95cdf19b858 redis:5.0.5-alpine],SizeBytes:29331594,},ContainerImage{Names:[quay.io/k8scsi/hostpathplugin@sha256:b4826e492fc1762fceaf9726f41575ca0a4567864d3d235da874818de18039de quay.io/k8scsi/hostpathplugin:v1.2.0-rc5],SizeBytes:28761497,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7],SizeBytes:22909487,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/echoserver@sha256:e9ba514b896cdf559eef8788b66c2c3ee55f3572df617647b4b0d8b6bf81cf19 gcr.io/kubernetes-e2e-test-images/echoserver:2.2],SizeBytes:21692741,},ContainerImage{Names:[quay.io/k8scsi/mock-driver@sha256:e0eed916b7d970bad2b7d9875f9ad16932f987f0f3d91ec5d86da68b0b5cc9d1 quay.io/k8scsi/mock-driver:v2.1.0],SizeBytes:16226335,},ContainerImage{Names:[nginx@sha256:485b610fefec7ff6c463ced9623314a04ed67e3945b9c08d7e53a47f6d108dc7 nginx:1.14-alpine],SizeBytes:16032814,},ContainerImage{Names:[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0],SizeBytes:15815995,},ContainerImage{Names:[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0],SizeBytes:14967303,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/dnsutils@sha256:2abeee84efb79c14d731966e034af33bf324d3b26ca28497555511ff094b3ddd gcr.io/kubernetes-e2e-test-images/dnsutils:1.1],SizeBytes:9349974,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nonewprivs@sha256:10066e9039219449fe3c81f38fe01928f87914150768ab81b62a468e51fa7411 gcr.io/kubernetes-e2e-test-images/nonewprivs:1.0],SizeBytes:6757579,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nautilus@sha256:33a732d4c42a266912a5091598a0f07653c9134db4b8d571690d8afd509e0bfc gcr.io/kubernetes-e2e-test-images/nautilus:1.0],SizeBytes:4753501,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/test-webserver@sha256:7f93d6e32798ff28bc6289254d0c2867fe2c849c8e46edc50f8624734309812e gcr.io/kubernetes-e2e-test-images/test-webserver:1.0],SizeBytes:4732240,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:4206620,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 gcr.io/kubernetes-e2e-test-images/mounttest:1.0],SizeBytes:1563521,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest-user@sha256:17319ca525ee003681fccf7e8c6b1b910ff4f49b653d939ac7f9b6e7c463933d gcr.io/kubernetes-e2e-test-images/mounttest-user:1.0],SizeBytes:1450451,},ContainerImage{Names:[busybox@sha256:1303dbf110c57f3edf68d9f5a16c082ec06c4cf7604831669faf2c712260b5a0 busybox:latest],SizeBytes:1219790,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29],SizeBytes:1154361,},ContainerImage{Names:[busybox@sha256:bbc3a03235220b170ba48a157dd097dd1379299370e1ed99ce976df0355d24f0 busybox:1.27],SizeBytes:1129289,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea k8s.gcr.io/pause-amd64:3.1 k8s.gcr.io/pause:3.1],SizeBytes:742472,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-3306^4],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-3306^4,DevicePath:,},},Config:nil,},} Nov 11 21:11:27.191: INFO: Logging kubelet events for node k8s-agentpool-35271922-vmss000001 Nov 11 21:11:27.262: INFO: Logging pods the kubelet thinks is on node k8s-agentpool-35271922-vmss000001 Nov 11 21:11:27.374: INFO: azure-ip-masq-agent-9427k started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:27.374: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:11:27.374: INFO: pod-subpath-test-configmap-lhw5 started at 2019-11-11 21:10:03 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:27.374: INFO: Container test-container-subpath-configmap-lhw5 ready: false, restart count 0 Nov 11 21:11:27.374: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-5xrt6 started at 2019-11-11 21:09:49 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:27.374: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: true, restart count 0 Nov 11 21:11:27.374: INFO: csi-mockplugin-resizer-0 started at 2019-11-11 21:08:30 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:27.374: INFO: Container csi-resizer ready: true, restart count 0 Nov 11 21:11:27.374: INFO: blobfuse-flexvol-installer-tvkwl started at 2019-11-11 20:48:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:27.374: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 Nov 11 21:11:27.374: INFO: tiller-deploy-7559b6b885-jsnn2 started at 2019-11-11 20:48:53 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:27.374: INFO: Container tiller ready: true, restart count 0 Nov 11 21:11:27.374: INFO: ss2-1 started at 2019-11-11 21:08:52 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:27.374: INFO: Container webserver ready: true, restart count 0 Nov 11 21:11:27.374: INFO: metrics-server-58ff8c5ddf-thxvp started at 2019-11-11 20:48:53 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:27.374: INFO: Container metrics-server ready: true, restart count 0 Nov 11 21:11:27.375: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-vzxn8 started at 2019-11-11 21:09:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:27.375: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: true, restart count 0 Nov 11 21:11:27.375: INFO: externalname-service-nbf4b started at 2019-11-11 21:10:25 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:27.375: INFO: Container externalname-service ready: true, restart count 0 Nov 11 21:11:27.375: INFO: replace-1573506660-vrzrd started at 2019-11-11 21:11:09 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:27.375: INFO: Container c ready: false, restart count 0 Nov 11 21:11:27.375: INFO: csi-mockplugin-attacher-0 started at 2019-11-11 21:07:07 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:27.375: INFO: Container csi-attacher ready: true, restart count 0 Nov 11 21:11:27.375: INFO: test-webserver-5659bd3a-d0a6-41c5-9e0b-bac115ac41d4 started at 2019-11-11 21:07:42 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:27.375: INFO: Container test-webserver ready: true, restart count 0 Nov 11 21:11:27.375: INFO: csi-mockplugin-attacher-0 started at 2019-11-11 21:08:29 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:27.375: INFO: Container csi-attacher ready: true, restart count 0 Nov 11 21:11:27.375: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-cbdrk started at 2019-11-11 21:09:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:27.375: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: true, restart count 0 Nov 11 21:11:27.375: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-m44k9 started at 2019-11-11 21:09:58 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:27.375: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: true, restart count 0 Nov 11 21:11:27.375: INFO: pod-ready started at 2019-11-11 21:10:19 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:27.375: INFO: Container pod-readiness-gate ready: true, restart count 0 Nov 11 21:11:27.375: INFO: httpd started at 2019-11-11 21:11:21 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:27.375: INFO: Container httpd ready: false, restart count 0 Nov 11 21:11:27.375: INFO: rs-k269h started at 2019-11-11 21:10:53 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:27.375: INFO: Container busybox ready: true, restart count 0 Nov 11 21:11:27.375: INFO: keyvault-flexvolume-2s2wq started at 2019-11-11 20:48:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:27.375: INFO: Container keyvault-flexvolume ready: true, restart count 0 Nov 11 21:11:27.375: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-sn584 started at 2019-11-11 21:09:53 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:27.375: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: true, restart count 0 Nov 11 21:11:27.375: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-gm5b8 started at 2019-11-11 21:10:00 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:27.375: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: true, restart count 0 Nov 11 21:11:27.375: INFO: test-runtimeclass-runtimeclass-5886-non-conflict-runtimeclgjbm4 started at 2019-11-11 21:11:23 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:27.375: INFO: Container test ready: false, restart count 0 Nov 11 21:11:27.375: INFO: pvc-volume-tester-mb6hz started at 2019-11-11 21:08:22 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:27.375: INFO: Container volume-tester ready: true, restart count 0 Nov 11 21:11:27.375: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-v8m7z started at 2019-11-11 21:09:50 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:27.375: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: true, restart count 0 Nov 11 21:11:27.375: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-nb9gn started at 2019-11-11 21:09:52 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:27.375: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: true, restart count 0 Nov 11 21:11:27.375: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-s2r9q started at 2019-11-11 21:10:02 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:27.375: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: true, restart count 0 Nov 11 21:11:27.375: INFO: rs-gkz92 started at 2019-11-11 21:10:52 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:27.375: INFO: Container busybox ready: true, restart count 0 Nov 11 21:11:27.375: INFO: csi-mockplugin-0 started at 2019-11-11 21:08:28 +0000 UTC (0+3 container statuses recorded) Nov 11 21:11:27.375: INFO: Container csi-provisioner ready: true, restart count 0 Nov 11 21:11:27.375: INFO: Container driver-registrar ready: true, restart count 0 Nov 11 21:11:27.375: INFO: Container mock ready: true, restart count 0 Nov 11 21:11:27.375: INFO: ss2-2 started at 2019-11-11 21:06:58 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:27.375: INFO: Container webserver ready: true, restart count 0 Nov 11 21:11:27.375: INFO: kube-proxy-sgvk2 started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:27.375: INFO: Container kube-proxy ready: true, restart count 0 Nov 11 21:11:27.375: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-xvlpv started at 2019-11-11 21:09:55 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:27.375: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: true, restart count 0 Nov 11 21:11:27.375: INFO: csi-mockplugin-0 started at 2019-11-11 21:07:06 +0000 UTC (0+3 container statuses recorded) Nov 11 21:11:27.375: INFO: Container csi-provisioner ready: true, restart count 0 Nov 11 21:11:27.375: INFO: Container driver-registrar ready: true, restart count 0 Nov 11 21:11:27.375: INFO: Container mock ready: true, restart count 0 W1111 21:11:27.431165 93351 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:11:30.643: INFO: Latency metrics for node k8s-agentpool-35271922-vmss000001 Nov 11 21:11:30.643: INFO: Logging node info for node k8s-master-35271922-vmss000000 Nov 11 21:11:30.698: INFO: Node Info: &Node{ObjectMeta:{k8s-master-35271922-vmss000000 /api/v1/nodes/k8s-master-35271922-vmss000000 f042d5b7-5ee8-4528-ae46-c997fef88c1c 39215 0 2019-11-11 20:48:07 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-35271922-vmss000000 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-35271922-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284363264 0} {<nil>} 7113636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6497931264 0} {<nil>} 6345636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:11:01 +0000 UTC,LastTransitionTime:2019-11-11 20:48:04 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:11:01 +0000 UTC,LastTransitionTime:2019-11-11 20:48:04 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:11:01 +0000 UTC,LastTransitionTime:2019-11-11 20:48:04 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:11:01 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.4,},NodeAddress{Type:Hostname,Address:k8s-master-35271922-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:8843d4f5cd0544d08ae501580db6a3a7,SystemUUID:41DCC090-746D-9E44-8888-8A77F0E0C51C,BootID:42d9eb02-ab49-4c2a-8791-7c5b2ac8f2bc,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:45e5b123e6188348715630acd2f7c1469c5b5181078ce90dc6f5399be4c17be1 k8sprow.azurecr.io/azure-cloud-controller-manager:1193984044413489154-9318bf1d],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:11:30.699: INFO: Logging kubelet events for node k8s-master-35271922-vmss000000 Nov 11 21:11:30.757: INFO: Logging pods the kubelet thinks is on node k8s-master-35271922-vmss000000 Nov 11 21:11:30.817: INFO: kube-apiserver-k8s-master-35271922-vmss000000 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:30.817: INFO: Container kube-apiserver ready: true, restart count 0 Nov 11 21:11:30.817: INFO: kube-controller-manager-k8s-master-35271922-vmss000000 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:30.817: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 11 21:11:30.817: INFO: kube-scheduler-k8s-master-35271922-vmss000000 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:30.817: INFO: Container kube-scheduler ready: true, restart count 0 Nov 11 21:11:30.817: INFO: azure-ip-masq-agent-k7l5b started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:30.817: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:11:30.817: INFO: kube-proxy-8hzbt started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:30.817: INFO: Container kube-proxy ready: true, restart count 0 Nov 11 21:11:30.817: INFO: cloud-controller-manager-k8s-master-35271922-vmss000000 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:30.817: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 11 21:11:30.817: INFO: kube-addon-manager-k8s-master-35271922-vmss000000 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:30.817: INFO: Container kube-addon-manager ready: true, restart count 0 W1111 21:11:30.873464 93351 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:11:31.010: INFO: Latency metrics for node k8s-master-35271922-vmss000000 Nov 11 21:11:31.011: INFO: Logging node info for node k8s-master-35271922-vmss000001 Nov 11 21:11:31.067: INFO: Node Info: &Node{ObjectMeta:{k8s-master-35271922-vmss000001 /api/v1/nodes/k8s-master-35271922-vmss000001 0ff06487-2e4c-4c07-a6f2-d77d1a47bbcd 39252 0 2019-11-11 20:48:39 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-35271922-vmss000001 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.6.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-35271922-vmss/virtualMachines/1,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.6.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284363264 0} {<nil>} 7113636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6497931264 0} {<nil>} 6345636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:49:01 +0000 UTC,LastTransitionTime:2019-11-11 20:49:01 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:11:03 +0000 UTC,LastTransitionTime:2019-11-11 20:48:32 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:11:03 +0000 UTC,LastTransitionTime:2019-11-11 20:48:32 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:11:03 +0000 UTC,LastTransitionTime:2019-11-11 20:48:32 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:11:03 +0000 UTC,LastTransitionTime:2019-11-11 20:48:39 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.5,},NodeAddress{Type:Hostname,Address:k8s-master-35271922-vmss000001,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:24b17940193c48e38f59d10d619fb718,SystemUUID:25FDA250-5862-9148-98C8-17FFDD874AEC,BootID:cdd7987a-6ece-464d-a177-21f53ae0b68b,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:45e5b123e6188348715630acd2f7c1469c5b5181078ce90dc6f5399be4c17be1 k8sprow.azurecr.io/azure-cloud-controller-manager:1193984044413489154-9318bf1d],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:11:31.067: INFO: Logging kubelet events for node k8s-master-35271922-vmss000001 Nov 11 21:11:31.126: INFO: Logging pods the kubelet thinks is on node k8s-master-35271922-vmss000001 Nov 11 21:11:31.188: INFO: kube-apiserver-k8s-master-35271922-vmss000001 started at 2019-11-11 20:48:26 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:31.188: INFO: Container kube-apiserver ready: true, restart count 0 Nov 11 21:11:31.188: INFO: kube-controller-manager-k8s-master-35271922-vmss000001 started at 2019-11-11 20:48:26 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:31.188: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 11 21:11:31.188: INFO: kube-scheduler-k8s-master-35271922-vmss000001 started at 2019-11-11 20:48:26 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:31.188: INFO: Container kube-scheduler ready: true, restart count 0 Nov 11 21:11:31.188: INFO: azure-ip-masq-agent-bm2t5 started at 2019-11-11 20:48:40 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:31.188: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:11:31.188: INFO: kube-proxy-5fcm6 started at 2019-11-11 20:48:40 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:31.188: INFO: Container kube-proxy ready: true, restart count 0 Nov 11 21:11:31.188: INFO: cloud-controller-manager-k8s-master-35271922-vmss000001 started at 2019-11-11 20:48:26 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:31.188: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 11 21:11:31.188: INFO: kube-addon-manager-k8s-master-35271922-vmss000001 started at 2019-11-11 20:48:26 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:31.188: INFO: Container kube-addon-manager ready: true, restart count 0 W1111 21:11:31.244799 93351 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:11:31.381: INFO: Latency metrics for node k8s-master-35271922-vmss000001 Nov 11 21:11:31.382: INFO: Logging node info for node k8s-master-35271922-vmss000002 Nov 11 21:11:31.436: INFO: Node Info: &Node{ObjectMeta:{k8s-master-35271922-vmss000002 /api/v1/nodes/k8s-master-35271922-vmss000002 7fedbabf-dce6-48d1-994b-628369e7bcb5 39241 0 2019-11-11 20:48:08 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-35271922-vmss000002 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.4.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-35271922-vmss/virtualMachines/2,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284363264 0} {<nil>} 7113636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6497931264 0} {<nil>} 6345636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:11:03 +0000 UTC,LastTransitionTime:2019-11-11 20:48:03 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:11:03 +0000 UTC,LastTransitionTime:2019-11-11 20:48:03 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:11:03 +0000 UTC,LastTransitionTime:2019-11-11 20:48:03 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:11:03 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.6,},NodeAddress{Type:Hostname,Address:k8s-master-35271922-vmss000002,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:5ecd4c723b8845b18d85d341ba062204,SystemUUID:9E772AEA-D3D7-084D-98ED-1C1A511168B4,BootID:a73a4655-81c3-42e2-9d20-ef7e3cff3664,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:45e5b123e6188348715630acd2f7c1469c5b5181078ce90dc6f5399be4c17be1 k8sprow.azurecr.io/azure-cloud-controller-manager:1193984044413489154-9318bf1d],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:11:31.437: INFO: Logging kubelet events for node k8s-master-35271922-vmss000002 Nov 11 21:11:31.495: INFO: Logging pods the kubelet thinks is on node k8s-master-35271922-vmss000002 Nov 11 21:11:31.558: INFO: kube-controller-manager-k8s-master-35271922-vmss000002 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:31.558: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 11 21:11:31.558: INFO: kube-scheduler-k8s-master-35271922-vmss000002 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:31.558: INFO: Container kube-scheduler ready: true, restart count 0 Nov 11 21:11:31.558: INFO: azure-ip-masq-agent-cmbqm started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:31.558: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:11:31.558: INFO: kube-proxy-t2sp7 started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:31.558: INFO: Container kube-proxy ready: true, restart count 0 Nov 11 21:11:31.558: INFO: cloud-controller-manager-k8s-master-35271922-vmss000002 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:31.558: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 11 21:11:31.558: INFO: kube-addon-manager-k8s-master-35271922-vmss000002 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:31.558: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 11 21:11:31.558: INFO: kube-apiserver-k8s-master-35271922-vmss000002 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:31.558: INFO: Container kube-apiserver ready: true, restart count 0 W1111 21:11:31.614925 93351 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:11:31.739: INFO: Latency metrics for node k8s-master-35271922-vmss000002 Nov 11 21:11:31.739: INFO: Logging node info for node k8s-master-35271922-vmss000003 Nov 11 21:11:31.795: INFO: Node Info: &Node{ObjectMeta:{k8s-master-35271922-vmss000003 /api/v1/nodes/k8s-master-35271922-vmss000003 d4d04c51-fc7b-4cdc-a1e3-69990a198568 39262 0 2019-11-11 20:48:10 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-35271922-vmss000003 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.5.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-35271922-vmss/virtualMachines/3,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.5.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284363264 0} {<nil>} 7113636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6497931264 0} {<nil>} 6345636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:11:04 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:11:04 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:11:04 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:11:04 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.7,},NodeAddress{Type:Hostname,Address:k8s-master-35271922-vmss000003,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:53117d11ac104f35951d19e4a4448f8f,SystemUUID:048B9870-F8AE-BC47-80D8-26703FED9760,BootID:fc64edd8-d9c5-4d6f-b9e7-c902146bc724,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:45e5b123e6188348715630acd2f7c1469c5b5181078ce90dc6f5399be4c17be1 k8sprow.azurecr.io/azure-cloud-controller-manager:1193984044413489154-9318bf1d],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:11:31.795: INFO: Logging kubelet events for node k8s-master-35271922-vmss000003 Nov 11 21:11:31.854: INFO: Logging pods the kubelet thinks is on node k8s-master-35271922-vmss000003 Nov 11 21:11:31.913: INFO: kube-addon-manager-k8s-master-35271922-vmss000003 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:31.913: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 11 21:11:31.913: INFO: kube-apiserver-k8s-master-35271922-vmss000003 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:31.913: INFO: Container kube-apiserver ready: true, restart count 0 Nov 11 21:11:31.914: INFO: kube-controller-manager-k8s-master-35271922-vmss000003 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:31.914: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 11 21:11:31.914: INFO: kube-scheduler-k8s-master-35271922-vmss000003 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:31.914: INFO: Container kube-scheduler ready: true, restart count 0 Nov 11 21:11:31.914: INFO: azure-ip-masq-agent-rs668 started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:31.914: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:11:31.914: INFO: kube-proxy-tqjkf started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:31.914: INFO: Container kube-proxy ready: true, restart count 0 Nov 11 21:11:31.914: INFO: cloud-controller-manager-k8s-master-35271922-vmss000003 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:31.914: INFO: Container cloud-controller-manager ready: true, restart count 0 W1111 21:11:31.970526 93351 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:11:32.097: INFO: Latency metrics for node k8s-master-35271922-vmss000003 Nov 11 21:11:32.097: INFO: Logging node info for node k8s-master-35271922-vmss000004 Nov 11 21:11:32.156: INFO: Node Info: &Node{ObjectMeta:{k8s-master-35271922-vmss000004 /api/v1/nodes/k8s-master-35271922-vmss000004 edf29ad2-2fc8-4411-ac05-e1bf8781e340 39311 0 2019-11-11 20:48:11 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-35271922-vmss000004 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-35271922-vmss/virtualMachines/4,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284363264 0} {<nil>} 7113636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6497931264 0} {<nil>} 6345636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:11:05 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:11:05 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:11:05 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:11:05 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.8,},NodeAddress{Type:Hostname,Address:k8s-master-35271922-vmss000004,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:3ecc95ca67e14e2e87659efc6707e3b4,SystemUUID:0649BB2F-7C03-5743-A309-9BABFF3DD11A,BootID:1256b8d0-bf8e-4490-a45d-beaf41e45a09,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:45e5b123e6188348715630acd2f7c1469c5b5181078ce90dc6f5399be4c17be1 k8sprow.azurecr.io/azure-cloud-controller-manager:1193984044413489154-9318bf1d],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:11:32.156: INFO: Logging kubelet events for node k8s-master-35271922-vmss000004 Nov 11 21:11:32.215: INFO: Logging pods the kubelet thinks is on node k8s-master-35271922-vmss000004 Nov 11 21:11:32.275: INFO: kube-apiserver-k8s-master-35271922-vmss000004 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:32.275: INFO: Container kube-apiserver ready: true, restart count 0 Nov 11 21:11:32.275: INFO: kube-controller-manager-k8s-master-35271922-vmss000004 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:32.275: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 11 21:11:32.275: INFO: kube-scheduler-k8s-master-35271922-vmss000004 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:32.275: INFO: Container kube-scheduler ready: true, restart count 0 Nov 11 21:11:32.275: INFO: azure-ip-masq-agent-trzsm started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:32.275: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:11:32.275: INFO: kube-proxy-zrmqq started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:32.275: INFO: Container kube-proxy ready: true, restart count 0 Nov 11 21:11:32.275: INFO: cloud-controller-manager-k8s-master-35271922-vmss000004 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:32.275: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 11 21:11:32.275: INFO: kube-addon-manager-k8s-master-35271922-vmss000004 started at 2019-11-11 20:48:19 +0000 UTC (0+1 container statuses recorded) Nov 11 21:11:32.275: INFO: Container kube-addon-manager ready: true, restart count 0 W1111 21:11:32.332049 93351 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:11:32.463: INFO: Latency metrics for node k8s-master-35271922-vmss000004 Nov 11 21:11:32.463: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "pods-4997" for this suite. Nov 11 21:14:56.694: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered Nov 11 21:14:58.579: INFO: namespace pods-4997 deletion completed in 3m26.059681115s
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[k8s\.io\]\s\[sig\-node\]\sPods\sExtended\s\[k8s\.io\]\sDelete\sGrace\sPeriod\sshould\sbe\ssubmitted\sand\sremoved\s\[Conformance\]$'
test/e2e/framework/framework.go:698 Nov 11 21:23:29.950: kubelet never observed the termination notice Unexpected error: <*errors.errorString | 0xc0000a10a0>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred test/e2e/node/pods.go:163from junit_20.xml
[BeforeEach] [k8s.io] [sig-node] Pods Extended test/e2e/framework/framework.go:151 �[1mSTEP�[0m: Creating a kubernetes client Nov 11 21:22:08.469: INFO: >>> kubeConfig: /workspace/aks147280421/kubeconfig/kubeconfig.westus2.json �[1mSTEP�[0m: Building a namespace api object, basename pods �[1mSTEP�[0m: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in pods-1821 �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace [BeforeEach] [k8s.io] Delete Grace Period test/e2e/node/pods.go:47 [It] should be submitted and removed [Conformance] test/e2e/framework/framework.go:698 �[1mSTEP�[0m: creating the pod �[1mSTEP�[0m: setting up selector �[1mSTEP�[0m: submitting the pod to kubernetes �[1mSTEP�[0m: verifying the pod is in kubernetes Nov 11 21:22:59.305: INFO: Asynchronously running '/usr/local/bin/kubectl kubectl --server=https://kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f.westus2.cloudapp.azure.com --kubeconfig=/workspace/aks147280421/kubeconfig/kubeconfig.westus2.json proxy -p 0' �[1mSTEP�[0m: deleting the pod gracefully �[1mSTEP�[0m: verifying the kubelet observed the termination notice Nov 11 21:23:29.950: FAIL: kubelet never observed the termination notice Unexpected error: <*errors.errorString | 0xc0000a10a0>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred [AfterEach] [k8s.io] [sig-node] Pods Extended test/e2e/framework/framework.go:152 �[1mSTEP�[0m: Collecting events from namespace "pods-1821". �[1mSTEP�[0m: Found 5 events. Nov 11 21:23:30.007: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for pod-submit-remove-4e755e27-38f7-4e59-a0f8-31aa18ae4fc4: {default-scheduler } Scheduled: Successfully assigned pods-1821/pod-submit-remove-4e755e27-38f7-4e59-a0f8-31aa18ae4fc4 to k8s-agentpool-35271922-vmss000001 Nov 11 21:23:30.007: INFO: At 2019-11-11 21:22:25 +0000 UTC - event for pod-submit-remove-4e755e27-38f7-4e59-a0f8-31aa18ae4fc4: {kubelet k8s-agentpool-35271922-vmss000001} Pulling: Pulling image "docker.io/library/nginx:1.14-alpine" Nov 11 21:23:30.007: INFO: At 2019-11-11 21:22:29 +0000 UTC - event for pod-submit-remove-4e755e27-38f7-4e59-a0f8-31aa18ae4fc4: {kubelet k8s-agentpool-35271922-vmss000001} Pulled: Successfully pulled image "docker.io/library/nginx:1.14-alpine" Nov 11 21:23:30.007: INFO: At 2019-11-11 21:22:34 +0000 UTC - event for pod-submit-remove-4e755e27-38f7-4e59-a0f8-31aa18ae4fc4: {kubelet k8s-agentpool-35271922-vmss000001} Created: Created container nginx Nov 11 21:23:30.007: INFO: At 2019-11-11 21:22:47 +0000 UTC - event for pod-submit-remove-4e755e27-38f7-4e59-a0f8-31aa18ae4fc4: {kubelet k8s-agentpool-35271922-vmss000001} Started: Started container nginx Nov 11 21:23:30.063: INFO: POD NODE PHASE GRACE CONDITIONS Nov 11 21:23:30.063: INFO: pod-submit-remove-4e755e27-38f7-4e59-a0f8-31aa18ae4fc4 k8s-agentpool-35271922-vmss000001 Running 30s [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:22:09 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:22:57 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:22:57 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:22:09 +0000 UTC }] Nov 11 21:23:30.063: INFO: Nov 11 21:23:30.215: INFO: Logging node info for node k8s-agentpool-35271922-vmss000000 Nov 11 21:23:30.270: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool-35271922-vmss000000 /api/v1/nodes/k8s-agentpool-35271922-vmss000000 04da751e-f23a-470e-b29d-e591113d7271 53507 0 2019-11-11 20:48:07 +0000 UTC <nil> <nil> map[agentpool:agentpool beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool-35271922-vmss000000 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: storageprofile:managed storagetier:Premium_LRS] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-2030":"k8s-agentpool-35271922-vmss000000","csi-hostpath-ephemeral-7436":"k8s-agentpool-35271922-vmss000000","csi-hostpath-provisioning-239":"k8s-agentpool-35271922-vmss000000","csi-hostpath-provisioning-3597":"k8s-agentpool-35271922-vmss000000","csi-hostpath-provisioning-9334":"k8s-agentpool-35271922-vmss000000","csi-hostpath-volume-5092":"k8s-agentpool-35271922-vmss000000","csi-hostpath-volume-expand-1449":"k8s-agentpool-35271922-vmss000000","csi-hostpath-volume-expand-5724":"k8s-agentpool-35271922-vmss000000","csi-hostpath-volume-expand-5832":"k8s-agentpool-35271922-vmss000000","csi-mock-csi-mock-volumes-1804":"csi-mock-csi-mock-volumes-1804","csi-mock-csi-mock-volumes-1829":"csi-mock-csi-mock-volumes-1829","csi-mock-csi-mock-volumes-2481":"csi-mock-csi-mock-volumes-2481","csi-mock-csi-mock-volumes-3755":"csi-mock-csi-mock-volumes-3755","csi-mock-csi-mock-volumes-763":"csi-mock-csi-mock-volumes-763","csi-mock-csi-mock-volumes-9721":"csi-mock-csi-mock-volumes-9721"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-35271922-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16797044736 0} {<nil>} 16403364Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16010612736 0} {<nil>} 15635364Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:22:55 +0000 UTC,LastTransitionTime:2019-11-11 20:47:58 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:22:55 +0000 UTC,LastTransitionTime:2019-11-11 20:47:58 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:22:55 +0000 UTC,LastTransitionTime:2019-11-11 20:47:58 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:22:55 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.248.0.4,},NodeAddress{Type:Hostname,Address:k8s-agentpool-35271922-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:84c06f3ee41a47aaa558c81f34711895,SystemUUID:9BB24974-637B-EE4C-A221-0F74BB743424,BootID:59f7eb0c-c09c-4eae-a14b-0710f9e14004,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[quay.io/kubernetes_incubator/nfs-provisioner@sha256:df762117e3c891f2d2ddff46ecb0776ba1f9f3c44cfd7739b0683bcd7a7954a8 quay.io/kubernetes_incubator/nfs-provisioner:v2.2.2],SizeBytes:391772778,},ContainerImage{Names:[gluster/glusterdynamic-provisioner@sha256:90067cb05a7d217651e84576935934fd9dcff8e6bcdcbaa416bbf36fcd09dbd1 gluster/glusterdynamic-provisioner:v1.0],SizeBytes:373281573,},ContainerImage{Names:[gcr.io/google-samples/gb-frontend@sha256:35cb427341429fac3df10ff74600ea73e8ec0754d78f9ce89e0b4f3d70d53ba6 gcr.io/google-samples/gb-frontend:v6],SizeBytes:373099368,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/volume/nfs@sha256:c2ad734346f608a5f7d69cfded93c4e8094069320657bd372d12ba21dea3ea71 gcr.io/kubernetes-e2e-test-images/volume/nfs:1.0],SizeBytes:225358913,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/jessie-dnsutils@sha256:ad583e33cb284f7ef046673809b146ec4053cda19b54a85d2b180a86169715eb gcr.io/kubernetes-e2e-test-images/jessie-dnsutils:1.0],SizeBytes:195659796,},ContainerImage{Names:[httpd@sha256:addd70e4ee83f3bc9a4c1c7c41e37927ba47faf639312fc936df3afad7926f5a httpd:2.4.39-alpine],SizeBytes:126894770,},ContainerImage{Names:[httpd@sha256:eb8ccf084cf3e80eece1add239effefd171eb39adbc154d33c14260d905d4060 httpd:2.4.38-alpine],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1],SizeBytes:121711221,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727 gcr.io/kubernetes-e2e-test-images/agnhost:2.6],SizeBytes:57345321,},ContainerImage{Names:[quay.io/k8scsi/csi-provisioner@sha256:0efcb424f1dde9b9fb11a1a14f2e48ab47e1c3f08bc3a929990dcfcb1f7ab34f quay.io/k8scsi/csi-provisioner:v1.4.0-rc1],SizeBytes:54431016,},ContainerImage{Names:[quay.io/k8scsi/csi-snapshotter@sha256:e3d3e742e32d00488fdb401045b9b1d033d7ca0ab6e760f77b24750fc95e5f70 quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1],SizeBytes:51703561,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0],SizeBytes:46226754,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:0aba670b4d9d6b2e720bbf575d733156c676b693ca26501235444490300db838 quay.io/k8scsi/csi-attacher:v1.1.0],SizeBytes:42839085,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:7d46fb6eb8b890dc546029d1565d502b4a1d974d33625c6ee2bc7991b77fc1a1 quay.io/k8scsi/csi-resizer:v0.2.0],SizeBytes:42817100,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:f315c9042e56def3c05c6b04fe79ec9da6d39ddc557ca365a76cf35964ea08b6 quay.io/k8scsi/csi-resizer:v0.1.0],SizeBytes:42623056,},ContainerImage{Names:[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0],SizeBytes:42488424,},ContainerImage{Names:[redis@sha256:50899ea1ceed33fa03232f3ac57578a424faa1742c1ac9c7a7bdb95cdf19b858 redis:5.0.5-alpine],SizeBytes:29331594,},ContainerImage{Names:[quay.io/k8scsi/hostpathplugin@sha256:b4826e492fc1762fceaf9726f41575ca0a4567864d3d235da874818de18039de quay.io/k8scsi/hostpathplugin:v1.2.0-rc5],SizeBytes:28761497,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7],SizeBytes:22909487,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/echoserver@sha256:e9ba514b896cdf559eef8788b66c2c3ee55f3572df617647b4b0d8b6bf81cf19 gcr.io/kubernetes-e2e-test-images/echoserver:2.2],SizeBytes:21692741,},ContainerImage{Names:[quay.io/k8scsi/mock-driver@sha256:e0eed916b7d970bad2b7d9875f9ad16932f987f0f3d91ec5d86da68b0b5cc9d1 quay.io/k8scsi/mock-driver:v2.1.0],SizeBytes:16226335,},ContainerImage{Names:[nginx@sha256:485b610fefec7ff6c463ced9623314a04ed67e3945b9c08d7e53a47f6d108dc7 nginx:1.14-alpine],SizeBytes:16032814,},ContainerImage{Names:[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0],SizeBytes:15815995,},ContainerImage{Names:[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0],SizeBytes:14967303,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/dnsutils@sha256:2abeee84efb79c14d731966e034af33bf324d3b26ca28497555511ff094b3ddd gcr.io/kubernetes-e2e-test-images/dnsutils:1.1],SizeBytes:9349974,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nautilus@sha256:33a732d4c42a266912a5091598a0f07653c9134db4b8d571690d8afd509e0bfc gcr.io/kubernetes-e2e-test-images/nautilus:1.0],SizeBytes:4753501,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/kitten@sha256:bcbc4875c982ab39aa7c4f6acf4a287f604e996d9f34a3fbda8c3d1a7457d1f6 gcr.io/kubernetes-e2e-test-images/kitten:1.0],SizeBytes:4747037,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/test-webserver@sha256:7f93d6e32798ff28bc6289254d0c2867fe2c849c8e46edc50f8624734309812e gcr.io/kubernetes-e2e-test-images/test-webserver:1.0],SizeBytes:4732240,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 gcr.io/kubernetes-e2e-test-images/mounttest:1.0],SizeBytes:1563521,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29],SizeBytes:1154361,},ContainerImage{Names:[busybox@sha256:bbc3a03235220b170ba48a157dd097dd1379299370e1ed99ce976df0355d24f0 busybox:1.27],SizeBytes:1129289,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea k8s.gcr.io/pause-amd64:3.1 k8s.gcr.io/pause:3.1],SizeBytes:742472,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-provisioning-239^65c5c89d-04c9-11ea-a9c8-000d3af7321e],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-provisioning-239^65c5c89d-04c9-11ea-a9c8-000d3af7321e,DevicePath:,},},Config:nil,},} Nov 11 21:23:30.271: INFO: Logging kubelet events for node k8s-agentpool-35271922-vmss000000 Nov 11 21:23:30.331: INFO: Logging pods the kubelet thinks is on node k8s-agentpool-35271922-vmss000000 Nov 11 21:23:30.449: INFO: inline-volume-tester-v8qbq started at 2019-11-11 21:14:53 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.449: INFO: Container csi-volume-tester ready: false, restart count 0 Nov 11 21:23:30.449: INFO: csi-hostpath-resizer-0 started at 2019-11-11 21:19:42 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.449: INFO: Container csi-resizer ready: true, restart count 0 Nov 11 21:23:30.449: INFO: csi-hostpath-resizer-0 started at 2019-11-11 21:18:13 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.449: INFO: Container csi-resizer ready: true, restart count 0 Nov 11 21:23:30.449: INFO: csi-snapshotter-0 started at 2019-11-11 21:18:13 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.449: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 11 21:23:30.449: INFO: blobfuse-flexvol-installer-dmnhw started at 2019-11-11 20:48:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.449: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 Nov 11 21:23:30.449: INFO: csi-hostpath-provisioner-0 started at 2019-11-11 21:18:13 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.449: INFO: Container csi-provisioner ready: true, restart count 0 Nov 11 21:23:30.449: INFO: coredns-87f5d796-6mz2l started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.449: INFO: Container coredns ready: true, restart count 0 Nov 11 21:23:30.449: INFO: pod-subpath-test-local-preprovisionedpv-pkvl started at 2019-11-11 21:22:15 +0000 UTC (1+1 container statuses recorded) Nov 11 21:23:30.449: INFO: Init container init-volume-local-preprovisionedpv-pkvl ready: true, restart count 0 Nov 11 21:23:30.449: INFO: Container test-container-subpath-local-preprovisionedpv-pkvl ready: false, restart count 0 Nov 11 21:23:30.449: INFO: csi-hostpath-attacher-0 started at 2019-11-11 21:19:23 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.449: INFO: Container csi-attacher ready: true, restart count 0 Nov 11 21:23:30.449: INFO: pod-subpath-test-local-preprovisionedpv-gjhc started at 2019-11-11 21:23:16 +0000 UTC (2+2 container statuses recorded) Nov 11 21:23:30.449: INFO: Init container init-volume-local-preprovisionedpv-gjhc ready: false, restart count 0 Nov 11 21:23:30.449: INFO: Init container test-init-subpath-local-preprovisionedpv-gjhc ready: false, restart count 0 Nov 11 21:23:30.449: INFO: Container test-container-subpath-local-preprovisionedpv-gjhc ready: false, restart count 0 Nov 11 21:23:30.449: INFO: Container test-container-volume-local-preprovisionedpv-gjhc ready: false, restart count 0 Nov 11 21:23:30.449: INFO: local-client started at 2019-11-11 21:22:30 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.449: INFO: Container local-client ready: true, restart count 0 Nov 11 21:23:30.449: INFO: local-client started at 2019-11-11 21:22:54 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.449: INFO: Container local-client ready: true, restart count 0 Nov 11 21:23:30.449: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:22:41 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.449: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:23:30.449: INFO: csi-snapshotter-0 started at 2019-11-11 21:19:44 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.449: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 11 21:23:30.449: INFO: csi-hostpathplugin-0 started at <nil> (0+0 container statuses recorded) Nov 11 21:23:30.449: INFO: kube-proxy-nwdpz started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.449: INFO: Container kube-proxy ready: true, restart count 0 Nov 11 21:23:30.449: INFO: csi-hostpath-attacher-0 started at 2019-11-11 21:14:46 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.449: INFO: Container csi-attacher ready: true, restart count 0 Nov 11 21:23:30.449: INFO: csi-hostpath-provisioner-0 started at 2019-11-11 21:19:23 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.449: INFO: Container csi-provisioner ready: true, restart count 0 Nov 11 21:23:30.449: INFO: csi-hostpath-resizer-0 started at 2019-11-11 21:19:24 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.449: INFO: Container csi-resizer ready: true, restart count 0 Nov 11 21:23:30.449: INFO: csi-hostpathplugin-0 started at 2019-11-11 21:21:22 +0000 UTC (0+3 container statuses recorded) Nov 11 21:23:30.449: INFO: Container hostpath ready: true, restart count 0 Nov 11 21:23:30.449: INFO: Container liveness-probe ready: true, restart count 0 Nov 11 21:23:30.449: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 11 21:23:30.449: INFO: azure-ip-masq-agent-c4mb2 started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.449: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:23:30.449: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:22:36 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.449: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:23:30.449: INFO: csi-hostpath-resizer-0 started at 2019-11-11 21:14:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.449: INFO: Container csi-resizer ready: true, restart count 0 Nov 11 21:23:30.449: INFO: security-context-ea38c576-290b-4c9e-8bd0-6063b39e4673 started at 2019-11-11 21:22:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.449: INFO: Container write-pod ready: true, restart count 0 Nov 11 21:23:30.449: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:20:33 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.449: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:23:30.449: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:20:38 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.449: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:23:30.449: INFO: local-client started at 2019-11-11 21:22:55 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.449: INFO: Container local-client ready: true, restart count 0 Nov 11 21:23:30.449: INFO: security-context-fc823105-2ca3-4045-883e-38bc1b4da1d1 started at 2019-11-11 21:23:13 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.449: INFO: Container write-pod ready: false, restart count 0 Nov 11 21:23:30.449: INFO: keyvault-flexvolume-zzp7p started at 2019-11-11 20:48:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.449: INFO: Container keyvault-flexvolume ready: true, restart count 0 Nov 11 21:23:30.449: INFO: csi-snapshotter-0 started at 2019-11-11 21:19:24 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.449: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 11 21:23:30.449: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:19:49 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.449: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:23:30.449: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:22:03 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.449: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:23:30.449: INFO: csi-hostpath-attacher-0 started at 2019-11-11 21:19:39 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.449: INFO: Container csi-attacher ready: true, restart count 0 Nov 11 21:23:30.449: INFO: csi-hostpath-provisioner-0 started at 2019-11-11 21:14:50 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.449: INFO: Container csi-provisioner ready: true, restart count 0 Nov 11 21:23:30.449: INFO: csi-hostpath-attacher-0 started at 2019-11-11 21:18:12 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.449: INFO: Container csi-attacher ready: true, restart count 0 Nov 11 21:23:30.449: INFO: kubernetes-dashboard-65966766b9-lv2x8 started at 2019-11-11 20:48:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.449: INFO: Container kubernetes-dashboard ready: true, restart count 0 Nov 11 21:23:30.449: INFO: csi-hostpathplugin-0 started at 2019-11-11 21:23:26 +0000 UTC (0+0 container statuses recorded) Nov 11 21:23:30.449: INFO: csi-snapshotter-0 started at 2019-11-11 21:14:52 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.449: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 11 21:23:30.449: INFO: security-context-499bab7c-5a6e-4429-bb4d-c8704ec76e42 started at 2019-11-11 21:23:28 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.449: INFO: Container write-pod ready: false, restart count 0 Nov 11 21:23:30.449: INFO: csi-hostpath-provisioner-0 started at 2019-11-11 21:19:42 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.449: INFO: Container csi-provisioner ready: true, restart count 0 Nov 11 21:23:30.449: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:21:34 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.449: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:23:30.449: INFO: pod-subpath-test-hostpathsymlink-pvr2 started at 2019-11-11 21:22:48 +0000 UTC (2+2 container statuses recorded) Nov 11 21:23:30.449: INFO: Init container init-volume-hostpathsymlink-pvr2 ready: true, restart count 0 Nov 11 21:23:30.449: INFO: Init container test-init-subpath-hostpathsymlink-pvr2 ready: true, restart count 0 Nov 11 21:23:30.449: INFO: Container test-container-subpath-hostpathsymlink-pvr2 ready: false, restart count 0 Nov 11 21:23:30.449: INFO: Container test-container-volume-hostpathsymlink-pvr2 ready: false, restart count 0 W1111 21:23:30.507273 93401 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:23:30.672: INFO: Latency metrics for node k8s-agentpool-35271922-vmss000000 Nov 11 21:23:30.672: INFO: Logging node info for node k8s-agentpool-35271922-vmss000001 Nov 11 21:23:30.727: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool-35271922-vmss000001 /api/v1/nodes/k8s-agentpool-35271922-vmss000001 af5ba0b6-4f7f-4468-9d3c-b3e116ca81ca 53831 0 2019-11-11 20:48:13 +0000 UTC <nil> <nil> map[agentpool:agentpool beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool-35271922-vmss000001 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: storageprofile:managed storagetier:Premium_LRS] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-8406":"k8s-agentpool-35271922-vmss000001","csi-hostpath-provisioning-2466":"k8s-agentpool-35271922-vmss000001","csi-hostpath-provisioning-265":"k8s-agentpool-35271922-vmss000001","csi-hostpath-provisioning-310":"k8s-agentpool-35271922-vmss000001","csi-hostpath-provisioning-753":"k8s-agentpool-35271922-vmss000001","csi-hostpath-volume-expand-3983":"k8s-agentpool-35271922-vmss000001","csi-mock-csi-mock-volumes-2781":"csi-mock-csi-mock-volumes-2781","csi-mock-csi-mock-volumes-2838":"csi-mock-csi-mock-volumes-2838","csi-mock-csi-mock-volumes-3306":"csi-mock-csi-mock-volumes-3306","csi-mock-csi-mock-volumes-6769":"csi-mock-csi-mock-volumes-6769","csi-mock-csi-mock-volumes-694":"csi-mock-csi-mock-volumes-694","csi-mock-csi-mock-volumes-797":"csi-mock-csi-mock-volumes-797"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-35271922-vmss/virtualMachines/1,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16797044736 0} {<nil>} 16403364Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16010612736 0} {<nil>} 15635364Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:23:20 +0000 UTC,LastTransitionTime:2019-11-11 20:48:13 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:23:20 +0000 UTC,LastTransitionTime:2019-11-11 20:48:13 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:23:20 +0000 UTC,LastTransitionTime:2019-11-11 20:48:13 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:23:20 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.248.0.5,},NodeAddress{Type:Hostname,Address:k8s-agentpool-35271922-vmss000001,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:1df8c22200b84e64b76150d232ecb929,SystemUUID:B03EE801-7A2D-7140-BE20-B2C844575A73,BootID:d32206f7-48b0-46c5-a542-39721566b456,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[quay.io/kubernetes_incubator/nfs-provisioner@sha256:df762117e3c891f2d2ddff46ecb0776ba1f9f3c44cfd7739b0683bcd7a7954a8 quay.io/kubernetes_incubator/nfs-provisioner:v2.2.2],SizeBytes:391772778,},ContainerImage{Names:[gcr.io/google-samples/gb-frontend@sha256:35cb427341429fac3df10ff74600ea73e8ec0754d78f9ce89e0b4f3d70d53ba6 gcr.io/google-samples/gb-frontend:v6],SizeBytes:373099368,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/volume/nfs@sha256:c2ad734346f608a5f7d69cfded93c4e8094069320657bd372d12ba21dea3ea71 gcr.io/kubernetes-e2e-test-images/volume/nfs:1.0],SizeBytes:225358913,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/jessie-dnsutils@sha256:ad583e33cb284f7ef046673809b146ec4053cda19b54a85d2b180a86169715eb gcr.io/kubernetes-e2e-test-images/jessie-dnsutils:1.0],SizeBytes:195659796,},ContainerImage{Names:[httpd@sha256:addd70e4ee83f3bc9a4c1c7c41e37927ba47faf639312fc936df3afad7926f5a httpd:2.4.39-alpine],SizeBytes:126894770,},ContainerImage{Names:[httpd@sha256:eb8ccf084cf3e80eece1add239effefd171eb39adbc154d33c14260d905d4060 httpd:2.4.38-alpine],SizeBytes:123781643,},ContainerImage{Names:[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0],SizeBytes:71821984,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/sample-apiserver@sha256:1bafcc6fb1aa990b487850adba9cadc020e42d7905aa8a30481182a477ba24b0 gcr.io/kubernetes-e2e-test-images/sample-apiserver:1.10],SizeBytes:61365829,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727 gcr.io/kubernetes-e2e-test-images/agnhost:2.6],SizeBytes:57345321,},ContainerImage{Names:[quay.io/k8scsi/csi-provisioner@sha256:0efcb424f1dde9b9fb11a1a14f2e48ab47e1c3f08bc3a929990dcfcb1f7ab34f quay.io/k8scsi/csi-provisioner:v1.4.0-rc1],SizeBytes:54431016,},ContainerImage{Names:[quay.io/k8scsi/csi-snapshotter@sha256:e3d3e742e32d00488fdb401045b9b1d033d7ca0ab6e760f77b24750fc95e5f70 quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1],SizeBytes:51703561,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0],SizeBytes:46226754,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:0aba670b4d9d6b2e720bbf575d733156c676b693ca26501235444490300db838 quay.io/k8scsi/csi-attacher:v1.1.0],SizeBytes:42839085,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:7d46fb6eb8b890dc546029d1565d502b4a1d974d33625c6ee2bc7991b77fc1a1 quay.io/k8scsi/csi-resizer:v0.2.0],SizeBytes:42817100,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:f315c9042e56def3c05c6b04fe79ec9da6d39ddc557ca365a76cf35964ea08b6 quay.io/k8scsi/csi-resizer:v0.1.0],SizeBytes:42623056,},ContainerImage{Names:[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1],SizeBytes:42541759,},ContainerImage{Names:[gcr.io/google-containers/debian-base@sha256:6966a0aedd7592c18ff2dd803c08bd85780ee19f5e3a2e7cf908a4cd837afcde gcr.io/google-containers/debian-base:0.4.1],SizeBytes:42323657,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nonroot@sha256:d4ede5c74517090b6686219059118ed178cf4620f5db8781b32f806bb1e7395b gcr.io/kubernetes-e2e-test-images/nonroot:1.0],SizeBytes:42321438,},ContainerImage{Names:[redis@sha256:50899ea1ceed33fa03232f3ac57578a424faa1742c1ac9c7a7bdb95cdf19b858 redis:5.0.5-alpine],SizeBytes:29331594,},ContainerImage{Names:[quay.io/k8scsi/hostpathplugin@sha256:b4826e492fc1762fceaf9726f41575ca0a4567864d3d235da874818de18039de quay.io/k8scsi/hostpathplugin:v1.2.0-rc5],SizeBytes:28761497,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7],SizeBytes:22909487,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/echoserver@sha256:e9ba514b896cdf559eef8788b66c2c3ee55f3572df617647b4b0d8b6bf81cf19 gcr.io/kubernetes-e2e-test-images/echoserver:2.2],SizeBytes:21692741,},ContainerImage{Names:[quay.io/k8scsi/mock-driver@sha256:e0eed916b7d970bad2b7d9875f9ad16932f987f0f3d91ec5d86da68b0b5cc9d1 quay.io/k8scsi/mock-driver:v2.1.0],SizeBytes:16226335,},ContainerImage{Names:[nginx@sha256:485b610fefec7ff6c463ced9623314a04ed67e3945b9c08d7e53a47f6d108dc7 nginx:1.14-alpine],SizeBytes:16032814,},ContainerImage{Names:[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0],SizeBytes:15815995,},ContainerImage{Names:[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0],SizeBytes:14967303,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/dnsutils@sha256:2abeee84efb79c14d731966e034af33bf324d3b26ca28497555511ff094b3ddd gcr.io/kubernetes-e2e-test-images/dnsutils:1.1],SizeBytes:9349974,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nonewprivs@sha256:10066e9039219449fe3c81f38fe01928f87914150768ab81b62a468e51fa7411 gcr.io/kubernetes-e2e-test-images/nonewprivs:1.0],SizeBytes:6757579,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nautilus@sha256:33a732d4c42a266912a5091598a0f07653c9134db4b8d571690d8afd509e0bfc gcr.io/kubernetes-e2e-test-images/nautilus:1.0],SizeBytes:4753501,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/kitten@sha256:bcbc4875c982ab39aa7c4f6acf4a287f604e996d9f34a3fbda8c3d1a7457d1f6 gcr.io/kubernetes-e2e-test-images/kitten:1.0],SizeBytes:4747037,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/test-webserver@sha256:7f93d6e32798ff28bc6289254d0c2867fe2c849c8e46edc50f8624734309812e gcr.io/kubernetes-e2e-test-images/test-webserver:1.0],SizeBytes:4732240,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:4206620,},ContainerImage{Names:[alpine@sha256:8421d9a84432575381bfabd248f1eb56f3aa21d9d7cd2511583c68c9b7511d10 alpine:3.7],SizeBytes:4206494,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 gcr.io/kubernetes-e2e-test-images/mounttest:1.0],SizeBytes:1563521,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest-user@sha256:17319ca525ee003681fccf7e8c6b1b910ff4f49b653d939ac7f9b6e7c463933d gcr.io/kubernetes-e2e-test-images/mounttest-user:1.0],SizeBytes:1450451,},ContainerImage{Names:[busybox@sha256:1303dbf110c57f3edf68d9f5a16c082ec06c4cf7604831669faf2c712260b5a0 busybox:latest],SizeBytes:1219790,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29],SizeBytes:1154361,},ContainerImage{Names:[busybox@sha256:bbc3a03235220b170ba48a157dd097dd1379299370e1ed99ce976df0355d24f0 busybox:1.27],SizeBytes:1129289,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea k8s.gcr.io/pause-amd64:3.1 k8s.gcr.io/pause:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:23:30.728: INFO: Logging kubelet events for node k8s-agentpool-35271922-vmss000001 Nov 11 21:23:30.787: INFO: Logging pods the kubelet thinks is on node k8s-agentpool-35271922-vmss000001 Nov 11 21:23:30.852: INFO: fail-once-local-fqqbs started at 2019-11-11 21:22:13 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.852: INFO: Container c ready: false, restart count 0 Nov 11 21:23:30.852: INFO: kube-proxy-sgvk2 started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.852: INFO: Container kube-proxy ready: true, restart count 0 Nov 11 21:23:30.852: INFO: externalsvc-8rj4l started at 2019-11-11 21:22:45 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.852: INFO: Container externalsvc ready: false, restart count 0 Nov 11 21:23:30.852: INFO: downwardapi-volume-2e624673-e8e2-4fc9-a204-b0d8280f6d29 started at 2019-11-11 21:23:25 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.852: INFO: Container client-container ready: false, restart count 0 Nov 11 21:23:30.852: INFO: csi-hostpath-attacher-0 started at 2019-11-11 21:23:26 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.852: INFO: Container csi-attacher ready: false, restart count 0 Nov 11 21:23:30.852: INFO: e2e-test-httpd-rc-8fca2dbdbed1f8f0dbbf980d1f6929b5-l4sb6 started at 2019-11-11 21:22:08 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.852: INFO: Container e2e-test-httpd-rc ready: true, restart count 0 Nov 11 21:23:30.852: INFO: csi-hostpathplugin-0 started at <nil> (0+0 container statuses recorded) Nov 11 21:23:30.852: INFO: externalsvc-vfqmb started at 2019-11-11 21:22:46 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.852: INFO: Container externalsvc ready: false, restart count 0 Nov 11 21:23:30.852: INFO: azure-ip-masq-agent-9427k started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.852: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:23:30.852: INFO: sample-apiserver-deployment-8447597c78-w4btv started at 2019-11-11 21:22:17 +0000 UTC (0+2 container statuses recorded) Nov 11 21:23:30.852: INFO: Container etcd ready: false, restart count 0 Nov 11 21:23:30.852: INFO: Container sample-apiserver ready: false, restart count 0 Nov 11 21:23:30.852: INFO: sample-crd-conversion-webhook-deployment-64d485d9bb-qrlw8 started at 2019-11-11 21:22:37 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.852: INFO: Container sample-crd-conversion-webhook ready: false, restart count 0 Nov 11 21:23:30.852: INFO: fail-once-local-m96ld started at 2019-11-11 21:20:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.852: INFO: Container c ready: false, restart count 1 Nov 11 21:23:30.852: INFO: blobfuse-flexvol-installer-tvkwl started at 2019-11-11 20:48:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.852: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 Nov 11 21:23:30.852: INFO: tiller-deploy-7559b6b885-jsnn2 started at 2019-11-11 20:48:53 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.852: INFO: Container tiller ready: true, restart count 0 Nov 11 21:23:30.852: INFO: busybox-3cc5f96b-0409-4e72-b3ab-eae766eb08d2 started at 2019-11-11 21:22:14 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.852: INFO: Container busybox ready: true, restart count 0 Nov 11 21:23:30.852: INFO: image-pull-teste9cbad06-1466-4ada-adc6-8eae74913eff started at 2019-11-11 21:22:24 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.852: INFO: Container image-pull-test ready: false, restart count 0 Nov 11 21:23:30.852: INFO: logs-generator started at 2019-11-11 21:22:05 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.852: INFO: Container logs-generator ready: false, restart count 0 Nov 11 21:23:30.852: INFO: fail-once-local-txr4g started at 2019-11-11 21:21:56 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.852: INFO: Container c ready: false, restart count 1 Nov 11 21:23:30.852: INFO: metrics-server-58ff8c5ddf-thxvp started at 2019-11-11 20:48:53 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.852: INFO: Container metrics-server ready: true, restart count 0 Nov 11 21:23:30.852: INFO: test-webserver-6b218e8a-73be-4386-8364-595903f1edca started at 2019-11-11 21:22:48 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.852: INFO: Container test-webserver ready: false, restart count 0 Nov 11 21:23:30.852: INFO: sample-webhook-deployment-86d95b659d-5fvsq started at 2019-11-11 21:23:23 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.852: INFO: Container sample-webhook ready: false, restart count 0 Nov 11 21:23:30.852: INFO: fail-once-local-gwsx5 started at 2019-11-11 21:20:58 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.852: INFO: Container c ready: false, restart count 1 Nov 11 21:23:30.852: INFO: nfs-server started at 2019-11-11 21:18:43 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.852: INFO: Container nfs-server ready: false, restart count 0 Nov 11 21:23:30.852: INFO: pod-submit-remove-4e755e27-38f7-4e59-a0f8-31aa18ae4fc4 started at 2019-11-11 21:22:09 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.852: INFO: Container nginx ready: true, restart count 0 Nov 11 21:23:30.852: INFO: backofflimit-jxgdp started at 2019-11-11 21:22:11 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.852: INFO: Container c ready: false, restart count 0 Nov 11 21:23:30.852: INFO: terminate-cmd-rpoff763d95d-70e1-4ce4-80b1-b33e29a44319 started at 2019-11-11 21:22:27 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.852: INFO: Container terminate-cmd-rpof ready: false, restart count 0 Nov 11 21:23:30.852: INFO: pfpod started at 2019-11-11 21:22:22 +0000 UTC (0+2 container statuses recorded) Nov 11 21:23:30.852: INFO: Container portforwardtester ready: false, restart count 0 Nov 11 21:23:30.852: INFO: Container readiness ready: false, restart count 0 Nov 11 21:23:30.852: INFO: keyvault-flexvolume-2s2wq started at 2019-11-11 20:48:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.852: INFO: Container keyvault-flexvolume ready: true, restart count 0 Nov 11 21:23:30.852: INFO: e2e-test-httpd-rc-6hxd6 started at 2019-11-11 21:22:06 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.852: INFO: Container e2e-test-httpd-rc ready: false, restart count 0 Nov 11 21:23:30.853: INFO: test-rollover-controller-lqzk7 started at 2019-11-11 21:22:47 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:30.853: INFO: Container httpd ready: false, restart count 0 W1111 21:23:30.909268 93401 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:23:31.060: INFO: Latency metrics for node k8s-agentpool-35271922-vmss000001 Nov 11 21:23:31.060: INFO: Logging node info for node k8s-master-35271922-vmss000000 Nov 11 21:23:31.116: INFO: Node Info: &Node{ObjectMeta:{k8s-master-35271922-vmss000000 /api/v1/nodes/k8s-master-35271922-vmss000000 f042d5b7-5ee8-4528-ae46-c997fef88c1c 53583 0 2019-11-11 20:48:07 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-35271922-vmss000000 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-35271922-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284363264 0} {<nil>} 7113636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6497931264 0} {<nil>} 6345636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:23:03 +0000 UTC,LastTransitionTime:2019-11-11 20:48:04 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:23:03 +0000 UTC,LastTransitionTime:2019-11-11 20:48:04 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:23:03 +0000 UTC,LastTransitionTime:2019-11-11 20:48:04 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:23:03 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.4,},NodeAddress{Type:Hostname,Address:k8s-master-35271922-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:8843d4f5cd0544d08ae501580db6a3a7,SystemUUID:41DCC090-746D-9E44-8888-8A77F0E0C51C,BootID:42d9eb02-ab49-4c2a-8791-7c5b2ac8f2bc,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:45e5b123e6188348715630acd2f7c1469c5b5181078ce90dc6f5399be4c17be1 k8sprow.azurecr.io/azure-cloud-controller-manager:1193984044413489154-9318bf1d],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:23:31.117: INFO: Logging kubelet events for node k8s-master-35271922-vmss000000 Nov 11 21:23:31.176: INFO: Logging pods the kubelet thinks is on node k8s-master-35271922-vmss000000 Nov 11 21:23:31.255: INFO: kube-addon-manager-k8s-master-35271922-vmss000000 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:31.255: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 11 21:23:31.255: INFO: kube-apiserver-k8s-master-35271922-vmss000000 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:31.255: INFO: Container kube-apiserver ready: true, restart count 0 Nov 11 21:23:31.255: INFO: kube-controller-manager-k8s-master-35271922-vmss000000 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:31.255: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 11 21:23:31.255: INFO: kube-scheduler-k8s-master-35271922-vmss000000 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:31.255: INFO: Container kube-scheduler ready: true, restart count 0 Nov 11 21:23:31.255: INFO: azure-ip-masq-agent-k7l5b started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:31.255: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:23:31.255: INFO: kube-proxy-8hzbt started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:31.255: INFO: Container kube-proxy ready: true, restart count 0 Nov 11 21:23:31.255: INFO: cloud-controller-manager-k8s-master-35271922-vmss000000 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:31.255: INFO: Container cloud-controller-manager ready: true, restart count 0 W1111 21:23:31.314063 93401 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:23:31.455: INFO: Latency metrics for node k8s-master-35271922-vmss000000 Nov 11 21:23:31.455: INFO: Logging node info for node k8s-master-35271922-vmss000001 Nov 11 21:23:31.516: INFO: Node Info: &Node{ObjectMeta:{k8s-master-35271922-vmss000001 /api/v1/nodes/k8s-master-35271922-vmss000001 0ff06487-2e4c-4c07-a6f2-d77d1a47bbcd 53606 0 2019-11-11 20:48:39 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-35271922-vmss000001 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.6.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-35271922-vmss/virtualMachines/1,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.6.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284363264 0} {<nil>} 7113636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6497931264 0} {<nil>} 6345636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:49:01 +0000 UTC,LastTransitionTime:2019-11-11 20:49:01 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:23:06 +0000 UTC,LastTransitionTime:2019-11-11 20:48:32 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:23:06 +0000 UTC,LastTransitionTime:2019-11-11 20:48:32 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:23:06 +0000 UTC,LastTransitionTime:2019-11-11 20:48:32 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:23:06 +0000 UTC,LastTransitionTime:2019-11-11 20:48:39 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.5,},NodeAddress{Type:Hostname,Address:k8s-master-35271922-vmss000001,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:24b17940193c48e38f59d10d619fb718,SystemUUID:25FDA250-5862-9148-98C8-17FFDD874AEC,BootID:cdd7987a-6ece-464d-a177-21f53ae0b68b,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:45e5b123e6188348715630acd2f7c1469c5b5181078ce90dc6f5399be4c17be1 k8sprow.azurecr.io/azure-cloud-controller-manager:1193984044413489154-9318bf1d],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:23:31.516: INFO: Logging kubelet events for node k8s-master-35271922-vmss000001 Nov 11 21:23:31.582: INFO: Logging pods the kubelet thinks is on node k8s-master-35271922-vmss000001 Nov 11 21:23:31.666: INFO: kube-controller-manager-k8s-master-35271922-vmss000001 started at 2019-11-11 20:48:26 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:31.666: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 11 21:23:31.666: INFO: kube-scheduler-k8s-master-35271922-vmss000001 started at 2019-11-11 20:48:26 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:31.666: INFO: Container kube-scheduler ready: true, restart count 0 Nov 11 21:23:31.666: INFO: azure-ip-masq-agent-bm2t5 started at 2019-11-11 20:48:40 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:31.666: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:23:31.666: INFO: kube-proxy-5fcm6 started at 2019-11-11 20:48:40 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:31.666: INFO: Container kube-proxy ready: true, restart count 0 Nov 11 21:23:31.666: INFO: cloud-controller-manager-k8s-master-35271922-vmss000001 started at 2019-11-11 20:48:26 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:31.666: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 11 21:23:31.666: INFO: kube-addon-manager-k8s-master-35271922-vmss000001 started at 2019-11-11 20:48:26 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:31.666: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 11 21:23:31.666: INFO: kube-apiserver-k8s-master-35271922-vmss000001 started at 2019-11-11 20:48:26 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:31.666: INFO: Container kube-apiserver ready: true, restart count 0 W1111 21:23:31.724060 93401 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:23:31.848: INFO: Latency metrics for node k8s-master-35271922-vmss000001 Nov 11 21:23:31.848: INFO: Logging node info for node k8s-master-35271922-vmss000002 Nov 11 21:23:31.908: INFO: Node Info: &Node{ObjectMeta:{k8s-master-35271922-vmss000002 /api/v1/nodes/k8s-master-35271922-vmss000002 7fedbabf-dce6-48d1-994b-628369e7bcb5 53598 0 2019-11-11 20:48:08 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-35271922-vmss000002 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.4.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-35271922-vmss/virtualMachines/2,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284363264 0} {<nil>} 7113636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6497931264 0} {<nil>} 6345636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:23:05 +0000 UTC,LastTransitionTime:2019-11-11 20:48:03 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:23:05 +0000 UTC,LastTransitionTime:2019-11-11 20:48:03 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:23:05 +0000 UTC,LastTransitionTime:2019-11-11 20:48:03 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:23:05 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.6,},NodeAddress{Type:Hostname,Address:k8s-master-35271922-vmss000002,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:5ecd4c723b8845b18d85d341ba062204,SystemUUID:9E772AEA-D3D7-084D-98ED-1C1A511168B4,BootID:a73a4655-81c3-42e2-9d20-ef7e3cff3664,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:45e5b123e6188348715630acd2f7c1469c5b5181078ce90dc6f5399be4c17be1 k8sprow.azurecr.io/azure-cloud-controller-manager:1193984044413489154-9318bf1d],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:23:31.909: INFO: Logging kubelet events for node k8s-master-35271922-vmss000002 Nov 11 21:23:31.974: INFO: Logging pods the kubelet thinks is on node k8s-master-35271922-vmss000002 Nov 11 21:23:32.067: INFO: kube-addon-manager-k8s-master-35271922-vmss000002 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:32.067: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 11 21:23:32.067: INFO: kube-apiserver-k8s-master-35271922-vmss000002 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:32.067: INFO: Container kube-apiserver ready: true, restart count 0 Nov 11 21:23:32.067: INFO: kube-controller-manager-k8s-master-35271922-vmss000002 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:32.067: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 11 21:23:32.067: INFO: kube-scheduler-k8s-master-35271922-vmss000002 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:32.067: INFO: Container kube-scheduler ready: true, restart count 0 Nov 11 21:23:32.067: INFO: azure-ip-masq-agent-cmbqm started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:32.067: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:23:32.067: INFO: kube-proxy-t2sp7 started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:32.067: INFO: Container kube-proxy ready: true, restart count 0 Nov 11 21:23:32.067: INFO: cloud-controller-manager-k8s-master-35271922-vmss000002 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:32.067: INFO: Container cloud-controller-manager ready: true, restart count 0 W1111 21:23:32.124811 93401 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:23:32.248: INFO: Latency metrics for node k8s-master-35271922-vmss000002 Nov 11 21:23:32.248: INFO: Logging node info for node k8s-master-35271922-vmss000003 Nov 11 21:23:32.305: INFO: Node Info: &Node{ObjectMeta:{k8s-master-35271922-vmss000003 /api/v1/nodes/k8s-master-35271922-vmss000003 d4d04c51-fc7b-4cdc-a1e3-69990a198568 53603 0 2019-11-11 20:48:10 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-35271922-vmss000003 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.5.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-35271922-vmss/virtualMachines/3,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.5.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284363264 0} {<nil>} 7113636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6497931264 0} {<nil>} 6345636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:23:06 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:23:06 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:23:06 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:23:06 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.7,},NodeAddress{Type:Hostname,Address:k8s-master-35271922-vmss000003,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:53117d11ac104f35951d19e4a4448f8f,SystemUUID:048B9870-F8AE-BC47-80D8-26703FED9760,BootID:fc64edd8-d9c5-4d6f-b9e7-c902146bc724,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:45e5b123e6188348715630acd2f7c1469c5b5181078ce90dc6f5399be4c17be1 k8sprow.azurecr.io/azure-cloud-controller-manager:1193984044413489154-9318bf1d],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:23:32.305: INFO: Logging kubelet events for node k8s-master-35271922-vmss000003 Nov 11 21:23:32.364: INFO: Logging pods the kubelet thinks is on node k8s-master-35271922-vmss000003 Nov 11 21:23:32.452: INFO: cloud-controller-manager-k8s-master-35271922-vmss000003 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:32.452: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 11 21:23:32.452: INFO: kube-addon-manager-k8s-master-35271922-vmss000003 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:32.452: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 11 21:23:32.452: INFO: kube-apiserver-k8s-master-35271922-vmss000003 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:32.452: INFO: Container kube-apiserver ready: true, restart count 0 Nov 11 21:23:32.452: INFO: kube-controller-manager-k8s-master-35271922-vmss000003 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:32.452: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 11 21:23:32.452: INFO: kube-scheduler-k8s-master-35271922-vmss000003 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:32.452: INFO: Container kube-scheduler ready: true, restart count 0 Nov 11 21:23:32.453: INFO: azure-ip-masq-agent-rs668 started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:32.453: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:23:32.453: INFO: kube-proxy-tqjkf started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:32.453: INFO: Container kube-proxy ready: true, restart count 0 W1111 21:23:32.523124 93401 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:23:32.654: INFO: Latency metrics for node k8s-master-35271922-vmss000003 Nov 11 21:23:32.654: INFO: Logging node info for node k8s-master-35271922-vmss000004 Nov 11 21:23:32.709: INFO: Node Info: &Node{ObjectMeta:{k8s-master-35271922-vmss000004 /api/v1/nodes/k8s-master-35271922-vmss000004 edf29ad2-2fc8-4411-ac05-e1bf8781e340 53617 0 2019-11-11 20:48:11 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-35271922-vmss000004 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-35271922-vmss/virtualMachines/4,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284363264 0} {<nil>} 7113636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6497931264 0} {<nil>} 6345636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:23:07 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:23:07 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:23:07 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:23:07 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.8,},NodeAddress{Type:Hostname,Address:k8s-master-35271922-vmss000004,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:3ecc95ca67e14e2e87659efc6707e3b4,SystemUUID:0649BB2F-7C03-5743-A309-9BABFF3DD11A,BootID:1256b8d0-bf8e-4490-a45d-beaf41e45a09,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:45e5b123e6188348715630acd2f7c1469c5b5181078ce90dc6f5399be4c17be1 k8sprow.azurecr.io/azure-cloud-controller-manager:1193984044413489154-9318bf1d],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:23:32.710: INFO: Logging kubelet events for node k8s-master-35271922-vmss000004 Nov 11 21:23:32.768: INFO: Logging pods the kubelet thinks is on node k8s-master-35271922-vmss000004 Nov 11 21:23:32.849: INFO: kube-apiserver-k8s-master-35271922-vmss000004 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:32.849: INFO: Container kube-apiserver ready: true, restart count 0 Nov 11 21:23:32.849: INFO: kube-controller-manager-k8s-master-35271922-vmss000004 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:32.849: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 11 21:23:32.849: INFO: kube-scheduler-k8s-master-35271922-vmss000004 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:32.849: INFO: Container kube-scheduler ready: true, restart count 0 Nov 11 21:23:32.849: INFO: azure-ip-masq-agent-trzsm started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:32.849: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:23:32.849: INFO: kube-proxy-zrmqq started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:32.849: INFO: Container kube-proxy ready: true, restart count 0 Nov 11 21:23:32.849: INFO: cloud-controller-manager-k8s-master-35271922-vmss000004 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:32.849: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 11 21:23:32.849: INFO: kube-addon-manager-k8s-master-35271922-vmss000004 started at 2019-11-11 20:48:19 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:32.849: INFO: Container kube-addon-manager ready: true, restart count 0 W1111 21:23:32.906946 93401 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:23:33.038: INFO: Latency metrics for node k8s-master-35271922-vmss000004 Nov 11 21:23:33.038: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "pods-1821" for this suite. Nov 11 21:27:57.272: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered Nov 11 21:27:59.149: INFO: namespace pods-1821 deletion completed in 4m26.05007952s
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[sig\-scheduling\]\sPreemptionExecutionPath\sruns\sReplicaSets\sto\sverify\spreemption\srunning\spath$'
test/e2e/scheduling/preemption.go:345 Nov 11 21:01:57.966: Unexpected error: <*errors.errorString | 0xc002112fb0>: { s: "replicaset \"rs-pod4\" never had desired number of .status.availableReplicas", } replicaset "rs-pod4" never had desired number of .status.availableReplicas occurred test/e2e/scheduling/preemption.go:510from junit_18.xml
[BeforeEach] [sig-scheduling] PreemptionExecutionPath test/e2e/framework/framework.go:151 �[1mSTEP�[0m: Creating a kubernetes client Nov 11 20:59:51.958: INFO: >>> kubeConfig: /workspace/aks147280421/kubeconfig/kubeconfig.westus2.json �[1mSTEP�[0m: Building a namespace api object, basename sched-preemption-path �[1mSTEP�[0m: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in sched-preemption-path-420 �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace [BeforeEach] [sig-scheduling] PreemptionExecutionPath test/e2e/scheduling/preemption.go:302 �[1mSTEP�[0m: Finding an available node �[1mSTEP�[0m: Trying to launch a pod without a label to get a node which can launch it. �[1mSTEP�[0m: Explicitly delete pod here to free the resource it takes. Nov 11 21:00:00.769: INFO: found a healthy node: k8s-agentpool-35271922-vmss000000 [It] runs ReplicaSets to verify preemption running path test/e2e/scheduling/preemption.go:345 Nov 11 21:00:57.714: INFO: pods created so far: map[rs-pod1-42m6m:{} rs-pod1-4hddx:{} rs-pod1-95b9h:{} rs-pod1-rtrss:{} rs-pod1-sr8jf:{} rs-pod2-bnbw4:{} rs-pod2-cgjvz:{} rs-pod2-dh4w6:{} rs-pod2-mwq8l:{} rs-pod3-cnnb8:{} rs-pod3-jfzbj:{} rs-pod3-q5q9h:{} rs-pod3-sgwnw:{}] Nov 11 21:00:57.714: INFO: length of pods created so far: 13 Nov 11 21:01:57.966: FAIL: Unexpected error: <*errors.errorString | 0xc002112fb0>: { s: "replicaset \"rs-pod4\" never had desired number of .status.availableReplicas", } replicaset "rs-pod4" never had desired number of .status.availableReplicas occurred [AfterEach] [sig-scheduling] PreemptionExecutionPath test/e2e/framework/framework.go:152 �[1mSTEP�[0m: Collecting events from namespace "sched-preemption-path-420". �[1mSTEP�[0m: Found 146 events. Nov 11 21:01:58.129: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-42m6m: {default-scheduler } Preempted: Preempted by sched-preemption-path-420/rs-pod4-lkc8x on node k8s-agentpool-35271922-vmss000000 Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-42m6m: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-42m6m: {default-scheduler } Scheduled: Successfully assigned sched-preemption-path-420/rs-pod1-42m6m to k8s-agentpool-35271922-vmss000000 Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-42m6m: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-4hddx: {default-scheduler } Scheduled: Successfully assigned sched-preemption-path-420/rs-pod1-4hddx to k8s-agentpool-35271922-vmss000000 Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-4hddx: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-4hddx: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-4hddx: {default-scheduler } Preempted: Preempted by sched-preemption-path-420/rs-pod4-lkc8x on node k8s-agentpool-35271922-vmss000000 Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-7nkm4: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-7nkm4: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-8zppd: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-8zppd: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-95b9h: {default-scheduler } Scheduled: Successfully assigned sched-preemption-path-420/rs-pod1-95b9h to k8s-agentpool-35271922-vmss000000 Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-95b9h: {default-scheduler } Preempted: Preempted by sched-preemption-path-420/rs-pod4-lkc8x on node k8s-agentpool-35271922-vmss000000 Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-95b9h: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-95b9h: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-h6c5q: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-h6c5q: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-rcdlr: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-rcdlr: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-rtrss: {default-scheduler } Scheduled: Successfully assigned sched-preemption-path-420/rs-pod1-rtrss to k8s-agentpool-35271922-vmss000000 Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-rtrss: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-rtrss: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-rtrss: {default-scheduler } Preempted: Preempted by sched-preemption-path-420/rs-pod4-lkc8x on node k8s-agentpool-35271922-vmss000000 Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-sd86t: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-sd86t: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-sr8jf: {default-scheduler } Scheduled: Successfully assigned sched-preemption-path-420/rs-pod1-sr8jf to k8s-agentpool-35271922-vmss000000 Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-sr8jf: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-sr8jf: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-sr8jf: {default-scheduler } Preempted: Preempted by sched-preemption-path-420/rs-pod4-lkc8x on node k8s-agentpool-35271922-vmss000000 Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod2-2vzpz: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod2-2vzpz: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod2-4pqdg: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod2-4pqdg: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod2-bnbw4: {default-scheduler } Scheduled: Successfully assigned sched-preemption-path-420/rs-pod2-bnbw4 to k8s-agentpool-35271922-vmss000000 Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod2-bnbw4: {default-scheduler } Preempted: Preempted by sched-preemption-path-420/rs-pod4-lkc8x on node k8s-agentpool-35271922-vmss000000 Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod2-cgjvz: {default-scheduler } Preempted: Preempted by sched-preemption-path-420/rs-pod4-lkc8x on node k8s-agentpool-35271922-vmss000000 Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod2-cgjvz: {default-scheduler } Scheduled: Successfully assigned sched-preemption-path-420/rs-pod2-cgjvz to k8s-agentpool-35271922-vmss000000 Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod2-dh4w6: {default-scheduler } Preempted: Preempted by sched-preemption-path-420/rs-pod4-lkc8x on node k8s-agentpool-35271922-vmss000000 Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod2-dh4w6: {default-scheduler } Scheduled: Successfully assigned sched-preemption-path-420/rs-pod2-dh4w6 to k8s-agentpool-35271922-vmss000000 Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod2-gxzwp: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod2-gxzwp: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod2-mwq8l: {default-scheduler } Scheduled: Successfully assigned sched-preemption-path-420/rs-pod2-mwq8l to k8s-agentpool-35271922-vmss000000 Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod2-mwq8l: {default-scheduler } Preempted: Preempted by sched-preemption-path-420/rs-pod4-lkc8x on node k8s-agentpool-35271922-vmss000000 Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod2-x774x: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod2-x774x: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod3-cnnb8: {default-scheduler } Scheduled: Successfully assigned sched-preemption-path-420/rs-pod3-cnnb8 to k8s-agentpool-35271922-vmss000000 Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod3-jfzbj: {default-scheduler } Scheduled: Successfully assigned sched-preemption-path-420/rs-pod3-jfzbj to k8s-agentpool-35271922-vmss000000 Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod3-q5q9h: {default-scheduler } Scheduled: Successfully assigned sched-preemption-path-420/rs-pod3-q5q9h to k8s-agentpool-35271922-vmss000000 Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod3-sgwnw: {default-scheduler } Scheduled: Successfully assigned sched-preemption-path-420/rs-pod3-sgwnw to k8s-agentpool-35271922-vmss000000 Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod4-lkc8x: {default-scheduler } Scheduled: Successfully assigned sched-preemption-path-420/rs-pod4-lkc8x to k8s-agentpool-35271922-vmss000000 Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod4-lkc8x: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod4-lkc8x: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 11 21:01:58.130: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for without-label: {default-scheduler } Scheduled: Successfully assigned sched-preemption-path-420/without-label to k8s-agentpool-35271922-vmss000000 Nov 11 21:01:58.130: INFO: At 2019-11-11 20:59:54 +0000 UTC - event for without-label: {kubelet k8s-agentpool-35271922-vmss000000} Pulled: Successfully pulled image "k8s.gcr.io/pause:3.1" Nov 11 21:01:58.130: INFO: At 2019-11-11 20:59:54 +0000 UTC - event for without-label: {kubelet k8s-agentpool-35271922-vmss000000} Pulling: Pulling image "k8s.gcr.io/pause:3.1" Nov 11 21:01:58.130: INFO: At 2019-11-11 20:59:55 +0000 UTC - event for without-label: {kubelet k8s-agentpool-35271922-vmss000000} Started: Started container without-label Nov 11 21:01:58.130: INFO: At 2019-11-11 20:59:55 +0000 UTC - event for without-label: {kubelet k8s-agentpool-35271922-vmss000000} Created: Created container without-label Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:00 +0000 UTC - event for without-label: {kubelet k8s-agentpool-35271922-vmss000000} Killing: Stopping container without-label Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:01 +0000 UTC - event for rs-pod1: {replicaset-controller } SuccessfulCreate: Created pod: rs-pod1-rtrss Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:01 +0000 UTC - event for rs-pod1: {replicaset-controller } SuccessfulCreate: Created pod: rs-pod1-42m6m Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:01 +0000 UTC - event for rs-pod1: {replicaset-controller } SuccessfulCreate: Created pod: rs-pod1-95b9h Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:01 +0000 UTC - event for rs-pod1: {replicaset-controller } SuccessfulCreate: Created pod: rs-pod1-4hddx Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:01 +0000 UTC - event for rs-pod1: {replicaset-controller } SuccessfulCreate: Created pod: rs-pod1-sr8jf Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:16 +0000 UTC - event for rs-pod1-42m6m: {kubelet k8s-agentpool-35271922-vmss000000} Pulling: Pulling image "k8s.gcr.io/pause:3.1" Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:16 +0000 UTC - event for rs-pod1-95b9h: {kubelet k8s-agentpool-35271922-vmss000000} Pulling: Pulling image "k8s.gcr.io/pause:3.1" Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:17 +0000 UTC - event for rs-pod1-rtrss: {kubelet k8s-agentpool-35271922-vmss000000} Pulling: Pulling image "k8s.gcr.io/pause:3.1" Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:17 +0000 UTC - event for rs-pod1-sr8jf: {kubelet k8s-agentpool-35271922-vmss000000} Pulling: Pulling image "k8s.gcr.io/pause:3.1" Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:18 +0000 UTC - event for rs-pod1-42m6m: {kubelet k8s-agentpool-35271922-vmss000000} Pulled: Successfully pulled image "k8s.gcr.io/pause:3.1" Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:18 +0000 UTC - event for rs-pod1-95b9h: {kubelet k8s-agentpool-35271922-vmss000000} Created: Created container pod1 Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:18 +0000 UTC - event for rs-pod1-95b9h: {kubelet k8s-agentpool-35271922-vmss000000} Pulled: Successfully pulled image "k8s.gcr.io/pause:3.1" Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:19 +0000 UTC - event for rs-pod1-42m6m: {kubelet k8s-agentpool-35271922-vmss000000} Started: Started container pod1 Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:19 +0000 UTC - event for rs-pod1-42m6m: {kubelet k8s-agentpool-35271922-vmss000000} Created: Created container pod1 Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:19 +0000 UTC - event for rs-pod1-95b9h: {kubelet k8s-agentpool-35271922-vmss000000} Started: Started container pod1 Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:19 +0000 UTC - event for rs-pod1-rtrss: {kubelet k8s-agentpool-35271922-vmss000000} Pulled: Successfully pulled image "k8s.gcr.io/pause:3.1" Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:19 +0000 UTC - event for rs-pod1-sr8jf: {kubelet k8s-agentpool-35271922-vmss000000} Created: Created container pod1 Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:19 +0000 UTC - event for rs-pod1-sr8jf: {kubelet k8s-agentpool-35271922-vmss000000} Pulled: Successfully pulled image "k8s.gcr.io/pause:3.1" Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:20 +0000 UTC - event for rs-pod1-4hddx: {kubelet k8s-agentpool-35271922-vmss000000} Pulling: Pulling image "k8s.gcr.io/pause:3.1" Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:20 +0000 UTC - event for rs-pod1-rtrss: {kubelet k8s-agentpool-35271922-vmss000000} Started: Started container pod1 Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:20 +0000 UTC - event for rs-pod1-rtrss: {kubelet k8s-agentpool-35271922-vmss000000} Created: Created container pod1 Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:20 +0000 UTC - event for rs-pod1-sr8jf: {kubelet k8s-agentpool-35271922-vmss000000} Started: Started container pod1 Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:21 +0000 UTC - event for rs-pod1-4hddx: {kubelet k8s-agentpool-35271922-vmss000000} Pulled: Successfully pulled image "k8s.gcr.io/pause:3.1" Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:21 +0000 UTC - event for rs-pod1-4hddx: {kubelet k8s-agentpool-35271922-vmss000000} Created: Created container pod1 Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:22 +0000 UTC - event for rs-pod1-4hddx: {kubelet k8s-agentpool-35271922-vmss000000} Started: Started container pod1 Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:29 +0000 UTC - event for rs-pod2: {replicaset-controller } SuccessfulCreate: Created pod: rs-pod2-bnbw4 Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:29 +0000 UTC - event for rs-pod2: {replicaset-controller } SuccessfulCreate: Created pod: rs-pod2-mwq8l Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:29 +0000 UTC - event for rs-pod2: {replicaset-controller } SuccessfulCreate: Created pod: rs-pod2-dh4w6 Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:29 +0000 UTC - event for rs-pod2: {replicaset-controller } SuccessfulCreate: Created pod: rs-pod2-cgjvz Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:33 +0000 UTC - event for rs-pod2-bnbw4: {kubelet k8s-agentpool-35271922-vmss000000} Pulling: Pulling image "k8s.gcr.io/pause:3.1" Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:34 +0000 UTC - event for rs-pod2-bnbw4: {kubelet k8s-agentpool-35271922-vmss000000} Pulled: Successfully pulled image "k8s.gcr.io/pause:3.1" Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:34 +0000 UTC - event for rs-pod2-cgjvz: {kubelet k8s-agentpool-35271922-vmss000000} Pulling: Pulling image "k8s.gcr.io/pause:3.1" Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:35 +0000 UTC - event for rs-pod2-bnbw4: {kubelet k8s-agentpool-35271922-vmss000000} Created: Created container pod2 Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:35 +0000 UTC - event for rs-pod2-bnbw4: {kubelet k8s-agentpool-35271922-vmss000000} Started: Started container pod2 Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:35 +0000 UTC - event for rs-pod2-cgjvz: {kubelet k8s-agentpool-35271922-vmss000000} Pulled: Successfully pulled image "k8s.gcr.io/pause:3.1" Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:35 +0000 UTC - event for rs-pod2-dh4w6: {kubelet k8s-agentpool-35271922-vmss000000} Pulling: Pulling image "k8s.gcr.io/pause:3.1" Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:35 +0000 UTC - event for rs-pod2-mwq8l: {kubelet k8s-agentpool-35271922-vmss000000} Pulling: Pulling image "k8s.gcr.io/pause:3.1" Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:36 +0000 UTC - event for rs-pod2-dh4w6: {kubelet k8s-agentpool-35271922-vmss000000} Pulled: Successfully pulled image "k8s.gcr.io/pause:3.1" Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:36 +0000 UTC - event for rs-pod2-mwq8l: {kubelet k8s-agentpool-35271922-vmss000000} Pulled: Successfully pulled image "k8s.gcr.io/pause:3.1" Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:37 +0000 UTC - event for rs-pod2-cgjvz: {kubelet k8s-agentpool-35271922-vmss000000} Started: Started container pod2 Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:37 +0000 UTC - event for rs-pod2-cgjvz: {kubelet k8s-agentpool-35271922-vmss000000} Created: Created container pod2 Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:37 +0000 UTC - event for rs-pod2-dh4w6: {kubelet k8s-agentpool-35271922-vmss000000} Created: Created container pod2 Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:37 +0000 UTC - event for rs-pod2-dh4w6: {kubelet k8s-agentpool-35271922-vmss000000} Started: Started container pod2 Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:37 +0000 UTC - event for rs-pod2-mwq8l: {kubelet k8s-agentpool-35271922-vmss000000} Started: Started container pod2 Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:37 +0000 UTC - event for rs-pod2-mwq8l: {kubelet k8s-agentpool-35271922-vmss000000} Created: Created container pod2 Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:45 +0000 UTC - event for rs-pod3: {replicaset-controller } SuccessfulCreate: Created pod: rs-pod3-q5q9h Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:45 +0000 UTC - event for rs-pod3: {replicaset-controller } SuccessfulCreate: Created pod: rs-pod3-cnnb8 Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:45 +0000 UTC - event for rs-pod3: {replicaset-controller } SuccessfulCreate: Created pod: rs-pod3-sgwnw Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:45 +0000 UTC - event for rs-pod3: {replicaset-controller } SuccessfulCreate: Created pod: rs-pod3-jfzbj Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:47 +0000 UTC - event for rs-pod3-cnnb8: {kubelet k8s-agentpool-35271922-vmss000000} Pulling: Pulling image "k8s.gcr.io/pause:3.1" Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:48 +0000 UTC - event for rs-pod3-cnnb8: {kubelet k8s-agentpool-35271922-vmss000000} Pulled: Successfully pulled image "k8s.gcr.io/pause:3.1" Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:48 +0000 UTC - event for rs-pod3-jfzbj: {kubelet k8s-agentpool-35271922-vmss000000} Pulling: Pulling image "k8s.gcr.io/pause:3.1" Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:48 +0000 UTC - event for rs-pod3-jfzbj: {kubelet k8s-agentpool-35271922-vmss000000} Pulled: Successfully pulled image "k8s.gcr.io/pause:3.1" Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:48 +0000 UTC - event for rs-pod3-sgwnw: {kubelet k8s-agentpool-35271922-vmss000000} Pulling: Pulling image "k8s.gcr.io/pause:3.1" Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:49 +0000 UTC - event for rs-pod3-cnnb8: {kubelet k8s-agentpool-35271922-vmss000000} Created: Created container pod3 Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:49 +0000 UTC - event for rs-pod3-cnnb8: {kubelet k8s-agentpool-35271922-vmss000000} Started: Started container pod3 Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:49 +0000 UTC - event for rs-pod3-jfzbj: {kubelet k8s-agentpool-35271922-vmss000000} Created: Created container pod3 Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:49 +0000 UTC - event for rs-pod3-q5q9h: {kubelet k8s-agentpool-35271922-vmss000000} Pulling: Pulling image "k8s.gcr.io/pause:3.1" Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:49 +0000 UTC - event for rs-pod3-q5q9h: {kubelet k8s-agentpool-35271922-vmss000000} Pulled: Successfully pulled image "k8s.gcr.io/pause:3.1" Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:49 +0000 UTC - event for rs-pod3-sgwnw: {kubelet k8s-agentpool-35271922-vmss000000} Pulled: Successfully pulled image "k8s.gcr.io/pause:3.1" Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:50 +0000 UTC - event for rs-pod3-jfzbj: {kubelet k8s-agentpool-35271922-vmss000000} Started: Started container pod3 Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:50 +0000 UTC - event for rs-pod3-q5q9h: {kubelet k8s-agentpool-35271922-vmss000000} Created: Created container pod3 Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:50 +0000 UTC - event for rs-pod3-sgwnw: {kubelet k8s-agentpool-35271922-vmss000000} Created: Created container pod3 Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:50 +0000 UTC - event for rs-pod3-sgwnw: {kubelet k8s-agentpool-35271922-vmss000000} Started: Started container pod3 Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:51 +0000 UTC - event for rs-pod3-q5q9h: {kubelet k8s-agentpool-35271922-vmss000000} Started: Started container pod3 Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:57 +0000 UTC - event for rs-pod1: {replicaset-controller } SuccessfulCreate: Created pod: rs-pod1-sd86t Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:57 +0000 UTC - event for rs-pod1-rtrss: {kubelet k8s-agentpool-35271922-vmss000000} Killing: Stopping container pod1 Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:57 +0000 UTC - event for rs-pod2: {replicaset-controller } SuccessfulCreate: Created pod: rs-pod2-2vzpz Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:57 +0000 UTC - event for rs-pod2: {replicaset-controller } SuccessfulCreate: Created pod: rs-pod2-gxzwp Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:57 +0000 UTC - event for rs-pod2: {replicaset-controller } SuccessfulCreate: Created pod: rs-pod2-x774x Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:57 +0000 UTC - event for rs-pod2: {replicaset-controller } SuccessfulCreate: Created pod: rs-pod2-4pqdg Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:57 +0000 UTC - event for rs-pod2-bnbw4: {kubelet k8s-agentpool-35271922-vmss000000} Killing: Stopping container pod2 Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:57 +0000 UTC - event for rs-pod2-cgjvz: {kubelet k8s-agentpool-35271922-vmss000000} Killing: Stopping container pod2 Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:57 +0000 UTC - event for rs-pod2-dh4w6: {kubelet k8s-agentpool-35271922-vmss000000} Killing: Stopping container pod2 Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:57 +0000 UTC - event for rs-pod2-mwq8l: {kubelet k8s-agentpool-35271922-vmss000000} Killing: Stopping container pod2 Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:57 +0000 UTC - event for rs-pod4: {replicaset-controller } SuccessfulCreate: Created pod: rs-pod4-lkc8x Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:58 +0000 UTC - event for rs-pod1: {replicaset-controller } SuccessfulCreate: (combined from similar events): Created pod: rs-pod1-rcdlr Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:58 +0000 UTC - event for rs-pod1: {replicaset-controller } SuccessfulCreate: Created pod: rs-pod1-8zppd Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:58 +0000 UTC - event for rs-pod1: {replicaset-controller } SuccessfulCreate: Created pod: rs-pod1-h6c5q Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:58 +0000 UTC - event for rs-pod1: {replicaset-controller } SuccessfulCreate: Created pod: rs-pod1-7nkm4 Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:58 +0000 UTC - event for rs-pod1-42m6m: {kubelet k8s-agentpool-35271922-vmss000000} Killing: Stopping container pod1 Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:58 +0000 UTC - event for rs-pod1-4hddx: {kubelet k8s-agentpool-35271922-vmss000000} Killing: Stopping container pod1 Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:58 +0000 UTC - event for rs-pod1-95b9h: {kubelet k8s-agentpool-35271922-vmss000000} Killing: Stopping container pod1 Nov 11 21:01:58.130: INFO: At 2019-11-11 21:00:58 +0000 UTC - event for rs-pod1-sr8jf: {kubelet k8s-agentpool-35271922-vmss000000} Killing: Stopping container pod1 Nov 11 21:01:58.130: INFO: At 2019-11-11 21:01:44 +0000 UTC - event for rs-pod4-lkc8x: {kubelet k8s-agentpool-35271922-vmss000000} Pulling: Pulling image "k8s.gcr.io/pause:3.1" Nov 11 21:01:58.130: INFO: At 2019-11-11 21:01:46 +0000 UTC - event for rs-pod4-lkc8x: {kubelet k8s-agentpool-35271922-vmss000000} Pulled: Successfully pulled image "k8s.gcr.io/pause:3.1" Nov 11 21:01:58.130: INFO: At 2019-11-11 21:01:53 +0000 UTC - event for rs-pod4-lkc8x: {kubelet k8s-agentpool-35271922-vmss000000} Created: Created container pod4 Nov 11 21:01:58.207: INFO: POD NODE PHASE GRACE CONDITIONS Nov 11 21:01:58.207: INFO: rs-pod1-7nkm4 Pending [{PodScheduled False 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:00:58 +0000 UTC Unschedulable 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu.}] Nov 11 21:01:58.207: INFO: rs-pod1-8zppd Pending [{PodScheduled False 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:00:58 +0000 UTC Unschedulable 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu.}] Nov 11 21:01:58.207: INFO: rs-pod1-h6c5q Pending [{PodScheduled False 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:00:58 +0000 UTC Unschedulable 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu.}] Nov 11 21:01:58.207: INFO: rs-pod1-rcdlr Pending [{PodScheduled False 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:00:58 +0000 UTC Unschedulable 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu.}] Nov 11 21:01:58.207: INFO: rs-pod1-sd86t Pending [{PodScheduled False 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:00:58 +0000 UTC Unschedulable 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu.}] Nov 11 21:01:58.207: INFO: rs-pod2-2vzpz Pending [{PodScheduled False 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:00:58 +0000 UTC Unschedulable 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu.}] Nov 11 21:01:58.207: INFO: rs-pod2-4pqdg Pending [{PodScheduled False 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:00:58 +0000 UTC Unschedulable 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu.}] Nov 11 21:01:58.207: INFO: rs-pod2-gxzwp Pending [{PodScheduled False 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:00:58 +0000 UTC Unschedulable 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu.}] Nov 11 21:01:58.207: INFO: rs-pod2-x774x Pending [{PodScheduled False 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:00:58 +0000 UTC Unschedulable 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu.}] Nov 11 21:01:58.208: INFO: rs-pod3-cnnb8 k8s-agentpool-35271922-vmss000000 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:00:45 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:00:50 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:00:50 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:00:45 +0000 UTC }] Nov 11 21:01:58.208: INFO: rs-pod3-jfzbj k8s-agentpool-35271922-vmss000000 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:00:45 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:00:51 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:00:51 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:00:45 +0000 UTC }] Nov 11 21:01:58.208: INFO: rs-pod3-q5q9h k8s-agentpool-35271922-vmss000000 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:00:45 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:00:52 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:00:52 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:00:45 +0000 UTC }] Nov 11 21:01:58.208: INFO: rs-pod3-sgwnw k8s-agentpool-35271922-vmss000000 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:00:45 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:00:52 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:00:52 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:00:45 +0000 UTC }] Nov 11 21:01:58.208: INFO: rs-pod4-lkc8x k8s-agentpool-35271922-vmss000000 Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:01:19 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:01:19 +0000 UTC ContainersNotReady containers with unready status: [pod4]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:01:19 +0000 UTC ContainersNotReady containers with unready status: [pod4]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:01:16 +0000 UTC }] Nov 11 21:01:58.208: INFO: Nov 11 21:01:58.332: INFO: Logging node info for node k8s-agentpool-35271922-vmss000000 Nov 11 21:01:58.388: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool-35271922-vmss000000 /api/v1/nodes/k8s-agentpool-35271922-vmss000000 04da751e-f23a-470e-b29d-e591113d7271 29085 0 2019-11-11 20:48:07 +0000 UTC <nil> <nil> map[agentpool:agentpool beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool-35271922-vmss000000 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: storageprofile:managed storagetier:Premium_LRS] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-2030":"k8s-agentpool-35271922-vmss000000","csi-hostpath-ephemeral-7436":"k8s-agentpool-35271922-vmss000000","csi-hostpath-provisioning-3597":"k8s-agentpool-35271922-vmss000000","csi-hostpath-provisioning-9334":"k8s-agentpool-35271922-vmss000000","csi-hostpath-volume-5092":"k8s-agentpool-35271922-vmss000000","csi-hostpath-volume-expand-1449":"k8s-agentpool-35271922-vmss000000","csi-hostpath-volume-expand-1922":"k8s-agentpool-35271922-vmss000000","csi-mock-csi-mock-volumes-1829":"csi-mock-csi-mock-volumes-1829","csi-mock-csi-mock-volumes-2481":"csi-mock-csi-mock-volumes-2481","csi-mock-csi-mock-volumes-9721":"csi-mock-csi-mock-volumes-9721"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-35271922-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},example.com/fakecpu: {{800 0} {<nil>} 800 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16797044736 0} {<nil>} 16403364Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},example.com/fakecpu: {{800 0} {<nil>} 800 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16010612736 0} {<nil>} 15635364Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:01:51 +0000 UTC,LastTransitionTime:2019-11-11 20:47:58 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:01:51 +0000 UTC,LastTransitionTime:2019-11-11 20:47:58 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:01:51 +0000 UTC,LastTransitionTime:2019-11-11 20:47:58 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:01:51 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.248.0.4,},NodeAddress{Type:Hostname,Address:k8s-agentpool-35271922-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:84c06f3ee41a47aaa558c81f34711895,SystemUUID:9BB24974-637B-EE4C-A221-0F74BB743424,BootID:59f7eb0c-c09c-4eae-a14b-0710f9e14004,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[quay.io/kubernetes_incubator/nfs-provisioner@sha256:df762117e3c891f2d2ddff46ecb0776ba1f9f3c44cfd7739b0683bcd7a7954a8 quay.io/kubernetes_incubator/nfs-provisioner:v2.2.2],SizeBytes:391772778,},ContainerImage{Names:[gluster/glusterdynamic-provisioner@sha256:90067cb05a7d217651e84576935934fd9dcff8e6bcdcbaa416bbf36fcd09dbd1 gluster/glusterdynamic-provisioner:v1.0],SizeBytes:373281573,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/volume/nfs@sha256:c2ad734346f608a5f7d69cfded93c4e8094069320657bd372d12ba21dea3ea71 gcr.io/kubernetes-e2e-test-images/volume/nfs:1.0],SizeBytes:225358913,},ContainerImage{Names:[httpd@sha256:addd70e4ee83f3bc9a4c1c7c41e37927ba47faf639312fc936df3afad7926f5a httpd:2.4.39-alpine],SizeBytes:126894770,},ContainerImage{Names:[httpd@sha256:eb8ccf084cf3e80eece1add239effefd171eb39adbc154d33c14260d905d4060 httpd:2.4.38-alpine],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1],SizeBytes:121711221,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727 gcr.io/kubernetes-e2e-test-images/agnhost:2.6],SizeBytes:57345321,},ContainerImage{Names:[quay.io/k8scsi/csi-provisioner@sha256:0efcb424f1dde9b9fb11a1a14f2e48ab47e1c3f08bc3a929990dcfcb1f7ab34f quay.io/k8scsi/csi-provisioner:v1.4.0-rc1],SizeBytes:54431016,},ContainerImage{Names:[quay.io/k8scsi/csi-snapshotter@sha256:e3d3e742e32d00488fdb401045b9b1d033d7ca0ab6e760f77b24750fc95e5f70 quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1],SizeBytes:51703561,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0],SizeBytes:46226754,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:0aba670b4d9d6b2e720bbf575d733156c676b693ca26501235444490300db838 quay.io/k8scsi/csi-attacher:v1.1.0],SizeBytes:42839085,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:7d46fb6eb8b890dc546029d1565d502b4a1d974d33625c6ee2bc7991b77fc1a1 quay.io/k8scsi/csi-resizer:v0.2.0],SizeBytes:42817100,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:f315c9042e56def3c05c6b04fe79ec9da6d39ddc557ca365a76cf35964ea08b6 quay.io/k8scsi/csi-resizer:v0.1.0],SizeBytes:42623056,},ContainerImage{Names:[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0],SizeBytes:42488424,},ContainerImage{Names:[quay.io/k8scsi/hostpathplugin@sha256:b4826e492fc1762fceaf9726f41575ca0a4567864d3d235da874818de18039de quay.io/k8scsi/hostpathplugin:v1.2.0-rc5],SizeBytes:28761497,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7],SizeBytes:22909487,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/echoserver@sha256:e9ba514b896cdf559eef8788b66c2c3ee55f3572df617647b4b0d8b6bf81cf19 gcr.io/kubernetes-e2e-test-images/echoserver:2.2],SizeBytes:21692741,},ContainerImage{Names:[quay.io/k8scsi/mock-driver@sha256:e0eed916b7d970bad2b7d9875f9ad16932f987f0f3d91ec5d86da68b0b5cc9d1 quay.io/k8scsi/mock-driver:v2.1.0],SizeBytes:16226335,},ContainerImage{Names:[nginx@sha256:485b610fefec7ff6c463ced9623314a04ed67e3945b9c08d7e53a47f6d108dc7 nginx:1.14-alpine],SizeBytes:16032814,},ContainerImage{Names:[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0],SizeBytes:15815995,},ContainerImage{Names:[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0],SizeBytes:14967303,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nautilus@sha256:33a732d4c42a266912a5091598a0f07653c9134db4b8d571690d8afd509e0bfc gcr.io/kubernetes-e2e-test-images/nautilus:1.0],SizeBytes:4753501,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 gcr.io/kubernetes-e2e-test-images/mounttest:1.0],SizeBytes:1563521,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29],SizeBytes:1154361,},ContainerImage{Names:[busybox@sha256:bbc3a03235220b170ba48a157dd097dd1379299370e1ed99ce976df0355d24f0 busybox:1.27],SizeBytes:1129289,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea k8s.gcr.io/pause-amd64:3.1 k8s.gcr.io/pause:3.1],SizeBytes:742472,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-volume-expand-1922^517c2929-04c6-11ea-b2d5-000d3af7321e],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-volume-expand-1922^517c2929-04c6-11ea-b2d5-000d3af7321e,DevicePath:,},},Config:nil,},} Nov 11 21:01:58.390: INFO: Logging kubelet events for node k8s-agentpool-35271922-vmss000000 Nov 11 21:01:58.483: INFO: Logging pods the kubelet thinks is on node k8s-agentpool-35271922-vmss000000 Nov 11 21:01:58.642: INFO: rs-pod3-sgwnw started at 2019-11-11 21:00:45 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:58.642: INFO: Container pod3 ready: true, restart count 0 Nov 11 21:01:58.642: INFO: e2e-test-httpd-pod started at 2019-11-11 21:00:58 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:58.642: INFO: Container e2e-test-httpd-pod ready: false, restart count 1 Nov 11 21:01:58.642: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:01:13 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:58.642: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:01:58.642: INFO: kubernetes-dashboard-65966766b9-lv2x8 started at 2019-11-11 20:48:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:58.643: INFO: Container kubernetes-dashboard ready: true, restart count 0 Nov 11 21:01:58.643: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:01:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:58.643: INFO: Container agnhost ready: false, restart count 0 Nov 11 21:01:58.643: INFO: pod-2 started at 2019-11-11 21:01:37 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:58.643: INFO: Container busybox ready: false, restart count 0 Nov 11 21:01:58.643: INFO: rs-pod3-jfzbj started at 2019-11-11 21:00:45 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:58.643: INFO: Container pod3 ready: true, restart count 0 Nov 11 21:01:58.643: INFO: csi-mockplugin-0 started at 2019-11-11 21:01:38 +0000 UTC (0+3 container statuses recorded) Nov 11 21:01:58.643: INFO: Container csi-provisioner ready: false, restart count 0 Nov 11 21:01:58.643: INFO: Container driver-registrar ready: false, restart count 0 Nov 11 21:01:58.643: INFO: Container mock ready: false, restart count 0 Nov 11 21:01:58.643: INFO: kube-proxy-nwdpz started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:58.643: INFO: Container kube-proxy ready: true, restart count 0 Nov 11 21:01:58.643: INFO: rs-pod3-cnnb8 started at 2019-11-11 21:00:45 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:58.643: INFO: Container pod3 ready: true, restart count 0 Nov 11 21:01:58.643: INFO: csi-mockplugin-attacher-0 started at 2019-11-11 21:01:39 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:58.643: INFO: Container csi-attacher ready: false, restart count 0 Nov 11 21:01:58.643: INFO: csi-hostpathplugin-0 started at 2019-11-11 21:00:32 +0000 UTC (0+3 container statuses recorded) Nov 11 21:01:58.643: INFO: Container hostpath ready: true, restart count 0 Nov 11 21:01:58.643: INFO: Container liveness-probe ready: true, restart count 0 Nov 11 21:01:58.643: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 11 21:01:58.643: INFO: exec-volume-test-emptydir-q8b2 started at 2019-11-11 21:01:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:58.643: INFO: Container exec-container-emptydir-q8b2 ready: false, restart count 0 Nov 11 21:01:58.643: INFO: csi-hostpath-provisioner-0 started at 2019-11-11 21:00:11 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:58.643: INFO: Container csi-provisioner ready: true, restart count 0 Nov 11 21:01:58.643: INFO: azure-ip-masq-agent-c4mb2 started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:58.643: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:01:58.643: INFO: affinity-nodeport-5t78p started at 2019-11-11 21:00:50 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:58.643: INFO: Container affinity-nodeport ready: true, restart count 0 Nov 11 21:01:58.643: INFO: ss2-0 started at 2019-11-11 21:01:16 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:58.643: INFO: Container webserver ready: true, restart count 0 Nov 11 21:01:58.643: INFO: security-context-31973928-eac5-43fa-9246-b4046edf6b67 started at 2019-11-11 21:01:54 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:58.643: INFO: Container write-pod ready: false, restart count 0 Nov 11 21:01:58.643: INFO: keyvault-flexvolume-zzp7p started at 2019-11-11 20:48:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:58.643: INFO: Container keyvault-flexvolume ready: true, restart count 0 Nov 11 21:01:58.643: INFO: blobfuse-flexvol-installer-dmnhw started at 2019-11-11 20:48:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:58.643: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 Nov 11 21:01:58.643: INFO: security-context-fd9cdef5-54d3-4b51-9842-462c3d1e8463 started at 2019-11-11 21:00:42 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:58.643: INFO: Container write-pod ready: true, restart count 0 Nov 11 21:01:58.643: INFO: csi-hostpath-attacher-0 started at 2019-11-11 21:00:10 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:58.643: INFO: Container csi-attacher ready: true, restart count 0 Nov 11 21:01:58.643: INFO: csi-hostpath-resizer-0 started at 2019-11-11 21:00:11 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:58.643: INFO: Container csi-resizer ready: true, restart count 0 Nov 11 21:01:58.643: INFO: httpd started at 2019-11-11 21:01:22 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:58.643: INFO: Container httpd ready: false, restart count 0 Nov 11 21:01:58.643: INFO: pod-subpath-test-hostpath-2dh5 started at 2019-11-11 21:01:26 +0000 UTC (1+1 container statuses recorded) Nov 11 21:01:58.643: INFO: Init container init-volume-hostpath-2dh5 ready: false, restart count 0 Nov 11 21:01:58.643: INFO: Container test-container-subpath-hostpath-2dh5 ready: false, restart count 0 Nov 11 21:01:58.643: INFO: rs-pod4-lkc8x started at 2019-11-11 21:01:19 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:58.643: INFO: Container pod4 ready: false, restart count 0 Nov 11 21:01:58.643: INFO: coredns-87f5d796-6mz2l started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:58.643: INFO: Container coredns ready: true, restart count 0 Nov 11 21:01:58.643: INFO: pod-subpath-test-local-preprovisionedpv-fpx4 started at 2019-11-11 21:01:18 +0000 UTC (2+1 container statuses recorded) Nov 11 21:01:58.643: INFO: Init container init-volume-local-preprovisionedpv-fpx4 ready: true, restart count 0 Nov 11 21:01:58.643: INFO: Init container test-init-volume-local-preprovisionedpv-fpx4 ready: false, restart count 0 Nov 11 21:01:58.643: INFO: Container test-container-subpath-local-preprovisionedpv-fpx4 ready: false, restart count 0 Nov 11 21:01:58.643: INFO: csi-snapshotter-0 started at 2019-11-11 21:00:11 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:58.643: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 11 21:01:58.643: INFO: rs-pod3-q5q9h started at 2019-11-11 21:00:45 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:58.643: INFO: Container pod3 ready: true, restart count 0 Nov 11 21:01:58.643: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:00:49 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:58.643: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:01:58.643: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:01:24 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:58.643: INFO: Container agnhost ready: false, restart count 0 W1111 21:01:58.711949 93347 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:01:58.897: INFO: Latency metrics for node k8s-agentpool-35271922-vmss000000 Nov 11 21:01:58.897: INFO: Logging node info for node k8s-agentpool-35271922-vmss000001 Nov 11 21:01:58.955: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool-35271922-vmss000001 /api/v1/nodes/k8s-agentpool-35271922-vmss000001 af5ba0b6-4f7f-4468-9d3c-b3e116ca81ca 28985 0 2019-11-11 20:48:13 +0000 UTC <nil> <nil> map[agentpool:agentpool beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool-35271922-vmss000001 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: storageprofile:managed storagetier:Premium_LRS] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-provisioning-2466":"k8s-agentpool-35271922-vmss000001","csi-hostpath-provisioning-265":"k8s-agentpool-35271922-vmss000001","csi-hostpath-provisioning-310":"k8s-agentpool-35271922-vmss000001","csi-hostpath-volume-expand-3983":"k8s-agentpool-35271922-vmss000001","csi-mock-csi-mock-volumes-2781":"csi-mock-csi-mock-volumes-2781","csi-mock-csi-mock-volumes-2838":"csi-mock-csi-mock-volumes-2838","csi-mock-csi-mock-volumes-797":"csi-mock-csi-mock-volumes-797"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-35271922-vmss/virtualMachines/1,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16797044736 0} {<nil>} 16403364Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16010612736 0} {<nil>} 15635364Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:01:46 +0000 UTC,LastTransitionTime:2019-11-11 20:48:13 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:01:46 +0000 UTC,LastTransitionTime:2019-11-11 20:48:13 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:01:46 +0000 UTC,LastTransitionTime:2019-11-11 20:48:13 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:01:46 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.248.0.5,},NodeAddress{Type:Hostname,Address:k8s-agentpool-35271922-vmss000001,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:1df8c22200b84e64b76150d232ecb929,SystemUUID:B03EE801-7A2D-7140-BE20-B2C844575A73,BootID:d32206f7-48b0-46c5-a542-39721566b456,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[quay.io/kubernetes_incubator/nfs-provisioner@sha256:df762117e3c891f2d2ddff46ecb0776ba1f9f3c44cfd7739b0683bcd7a7954a8 quay.io/kubernetes_incubator/nfs-provisioner:v2.2.2],SizeBytes:391772778,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/volume/nfs@sha256:c2ad734346f608a5f7d69cfded93c4e8094069320657bd372d12ba21dea3ea71 gcr.io/kubernetes-e2e-test-images/volume/nfs:1.0],SizeBytes:225358913,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/jessie-dnsutils@sha256:ad583e33cb284f7ef046673809b146ec4053cda19b54a85d2b180a86169715eb gcr.io/kubernetes-e2e-test-images/jessie-dnsutils:1.0],SizeBytes:195659796,},ContainerImage{Names:[httpd@sha256:addd70e4ee83f3bc9a4c1c7c41e37927ba47faf639312fc936df3afad7926f5a httpd:2.4.39-alpine],SizeBytes:126894770,},ContainerImage{Names:[httpd@sha256:eb8ccf084cf3e80eece1add239effefd171eb39adbc154d33c14260d905d4060 httpd:2.4.38-alpine],SizeBytes:123781643,},ContainerImage{Names:[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0],SizeBytes:71821984,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727 gcr.io/kubernetes-e2e-test-images/agnhost:2.6],SizeBytes:57345321,},ContainerImage{Names:[quay.io/k8scsi/csi-provisioner@sha256:0efcb424f1dde9b9fb11a1a14f2e48ab47e1c3f08bc3a929990dcfcb1f7ab34f quay.io/k8scsi/csi-provisioner:v1.4.0-rc1],SizeBytes:54431016,},ContainerImage{Names:[quay.io/k8scsi/csi-snapshotter@sha256:e3d3e742e32d00488fdb401045b9b1d033d7ca0ab6e760f77b24750fc95e5f70 quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1],SizeBytes:51703561,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0],SizeBytes:46226754,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:0aba670b4d9d6b2e720bbf575d733156c676b693ca26501235444490300db838 quay.io/k8scsi/csi-attacher:v1.1.0],SizeBytes:42839085,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:7d46fb6eb8b890dc546029d1565d502b4a1d974d33625c6ee2bc7991b77fc1a1 quay.io/k8scsi/csi-resizer:v0.2.0],SizeBytes:42817100,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:f315c9042e56def3c05c6b04fe79ec9da6d39ddc557ca365a76cf35964ea08b6 quay.io/k8scsi/csi-resizer:v0.1.0],SizeBytes:42623056,},ContainerImage{Names:[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1],SizeBytes:42541759,},ContainerImage{Names:[gcr.io/google-containers/debian-base@sha256:6966a0aedd7592c18ff2dd803c08bd85780ee19f5e3a2e7cf908a4cd837afcde gcr.io/google-containers/debian-base:0.4.1],SizeBytes:42323657,},ContainerImage{Names:[redis@sha256:50899ea1ceed33fa03232f3ac57578a424faa1742c1ac9c7a7bdb95cdf19b858 redis:5.0.5-alpine],SizeBytes:29331594,},ContainerImage{Names:[quay.io/k8scsi/hostpathplugin@sha256:b4826e492fc1762fceaf9726f41575ca0a4567864d3d235da874818de18039de quay.io/k8scsi/hostpathplugin:v1.2.0-rc5],SizeBytes:28761497,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7],SizeBytes:22909487,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/echoserver@sha256:e9ba514b896cdf559eef8788b66c2c3ee55f3572df617647b4b0d8b6bf81cf19 gcr.io/kubernetes-e2e-test-images/echoserver:2.2],SizeBytes:21692741,},ContainerImage{Names:[quay.io/k8scsi/mock-driver@sha256:e0eed916b7d970bad2b7d9875f9ad16932f987f0f3d91ec5d86da68b0b5cc9d1 quay.io/k8scsi/mock-driver:v2.1.0],SizeBytes:16226335,},ContainerImage{Names:[nginx@sha256:485b610fefec7ff6c463ced9623314a04ed67e3945b9c08d7e53a47f6d108dc7 nginx:1.14-alpine],SizeBytes:16032814,},ContainerImage{Names:[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0],SizeBytes:15815995,},ContainerImage{Names:[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0],SizeBytes:14967303,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/dnsutils@sha256:2abeee84efb79c14d731966e034af33bf324d3b26ca28497555511ff094b3ddd gcr.io/kubernetes-e2e-test-images/dnsutils:1.1],SizeBytes:9349974,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nonewprivs@sha256:10066e9039219449fe3c81f38fe01928f87914150768ab81b62a468e51fa7411 gcr.io/kubernetes-e2e-test-images/nonewprivs:1.0],SizeBytes:6757579,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nautilus@sha256:33a732d4c42a266912a5091598a0f07653c9134db4b8d571690d8afd509e0bfc gcr.io/kubernetes-e2e-test-images/nautilus:1.0],SizeBytes:4753501,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/test-webserver@sha256:7f93d6e32798ff28bc6289254d0c2867fe2c849c8e46edc50f8624734309812e gcr.io/kubernetes-e2e-test-images/test-webserver:1.0],SizeBytes:4732240,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:4206620,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 gcr.io/kubernetes-e2e-test-images/mounttest:1.0],SizeBytes:1563521,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest-user@sha256:17319ca525ee003681fccf7e8c6b1b910ff4f49b653d939ac7f9b6e7c463933d gcr.io/kubernetes-e2e-test-images/mounttest-user:1.0],SizeBytes:1450451,},ContainerImage{Names:[busybox@sha256:1303dbf110c57f3edf68d9f5a16c082ec06c4cf7604831669faf2c712260b5a0 busybox:latest],SizeBytes:1219790,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29],SizeBytes:1154361,},ContainerImage{Names:[busybox@sha256:bbc3a03235220b170ba48a157dd097dd1379299370e1ed99ce976df0355d24f0 busybox:1.27],SizeBytes:1129289,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea k8s.gcr.io/pause-amd64:3.1 k8s.gcr.io/pause:3.1],SizeBytes:742472,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-2838^4 kubernetes.io/csi/csi-mock-csi-mock-volumes-797^4],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-797^4,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-2838^4,DevicePath:,},},Config:nil,},} Nov 11 21:01:58.955: INFO: Logging kubelet events for node k8s-agentpool-35271922-vmss000001 Nov 11 21:01:59.015: INFO: Logging pods the kubelet thinks is on node k8s-agentpool-35271922-vmss000001 Nov 11 21:01:59.077: INFO: hostexec-k8s-agentpool-35271922-vmss000001 started at 2019-11-11 21:00:49 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:59.078: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:01:59.078: INFO: pod-1 started at 2019-11-11 21:01:37 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:59.078: INFO: Container busybox ready: false, restart count 0 Nov 11 21:01:59.078: INFO: busybox-4e763c71-d074-4e3a-aa4b-b476cb4de9c4 started at 2019-11-11 20:58:41 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:59.078: INFO: Container busybox ready: true, restart count 0 Nov 11 21:01:59.078: INFO: pod-subpath-test-local-preprovisionedpv-nsk9 started at 2019-11-11 21:01:29 +0000 UTC (2+1 container statuses recorded) Nov 11 21:01:59.078: INFO: Init container init-volume-local-preprovisionedpv-nsk9 ready: false, restart count 0 Nov 11 21:01:59.078: INFO: Init container test-init-subpath-local-preprovisionedpv-nsk9 ready: false, restart count 0 Nov 11 21:01:59.078: INFO: Container test-container-subpath-local-preprovisionedpv-nsk9 ready: false, restart count 0 Nov 11 21:01:59.078: INFO: pod-128072fd-7323-45c3-b5c2-87ef342cece3 started at 2019-11-11 21:01:35 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:59.078: INFO: Container test-container ready: false, restart count 0 Nov 11 21:01:59.078: INFO: execpod-affinitywjq89 started at 2019-11-11 21:01:05 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:59.078: INFO: Container agnhost-pause ready: true, restart count 0 Nov 11 21:01:59.078: INFO: sample-webhook-deployment-86d95b659d-pc6pk started at 2019-11-11 21:01:45 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:59.078: INFO: Container sample-webhook ready: false, restart count 0 Nov 11 21:01:59.078: INFO: azure-ip-masq-agent-9427k started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:59.078: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:01:59.078: INFO: csi-mockplugin-attacher-0 started at 2019-11-11 21:00:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:59.078: INFO: Container csi-attacher ready: true, restart count 0 Nov 11 21:01:59.078: INFO: liveness-4fac5e1d-195c-4756-b1b7-9d4a086a8ac1 started at 2019-11-11 20:59:50 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:59.078: INFO: Container liveness ready: false, restart count 4 Nov 11 21:01:59.078: INFO: keyvault-flexvolume-2s2wq started at 2019-11-11 20:48:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:59.078: INFO: Container keyvault-flexvolume ready: true, restart count 0 Nov 11 21:01:59.078: INFO: pod-projected-secrets-3ed43776-db9d-45ee-8b77-5efdc6ff4b5a started at 2019-11-11 21:01:43 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:59.078: INFO: Container secret-volume-test ready: false, restart count 0 Nov 11 21:01:59.078: INFO: ss2-1 started at 2019-11-11 21:01:47 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:59.078: INFO: Container webserver ready: false, restart count 0 Nov 11 21:01:59.078: INFO: hostexec-k8s-agentpool-35271922-vmss000001 started at 2019-11-11 21:00:58 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:59.078: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:01:59.078: INFO: csi-mockplugin-resizer-0 started at 2019-11-11 21:00:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:59.078: INFO: Container csi-resizer ready: true, restart count 0 Nov 11 21:01:59.078: INFO: blobfuse-flexvol-installer-tvkwl started at 2019-11-11 20:48:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:59.078: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 Nov 11 21:01:59.078: INFO: tiller-deploy-7559b6b885-jsnn2 started at 2019-11-11 20:48:53 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:59.078: INFO: Container tiller ready: true, restart count 0 Nov 11 21:01:59.078: INFO: pvc-volume-tester-w7727 started at 2019-11-11 21:01:28 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:59.078: INFO: Container volume-tester ready: false, restart count 0 Nov 11 21:01:59.078: INFO: httpd started at 2019-11-11 21:01:56 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:59.078: INFO: Container httpd ready: false, restart count 0 Nov 11 21:01:59.078: INFO: pod-handle-http-request started at 2019-11-11 21:00:48 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:59.078: INFO: Container pod-handle-http-request ready: true, restart count 0 Nov 11 21:01:59.078: INFO: recycler-for-nfs-2mk8g started at 2019-11-11 20:57:53 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:59.078: INFO: Container pv-recycler ready: true, restart count 0 Nov 11 21:01:59.078: INFO: kube-proxy-sgvk2 started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:59.078: INFO: Container kube-proxy ready: true, restart count 0 Nov 11 21:01:59.078: INFO: metrics-server-58ff8c5ddf-thxvp started at 2019-11-11 20:48:53 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:59.078: INFO: Container metrics-server ready: true, restart count 0 Nov 11 21:01:59.078: INFO: pod-0 started at 2019-11-11 21:01:36 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:59.078: INFO: Container busybox ready: false, restart count 0 Nov 11 21:01:59.078: INFO: hostexec-k8s-agentpool-35271922-vmss000001 started at 2019-11-11 21:01:46 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:59.078: INFO: Container agnhost ready: false, restart count 0 Nov 11 21:01:59.078: INFO: hostexec-k8s-agentpool-35271922-vmss000001 started at 2019-11-11 21:01:52 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:59.078: INFO: Container agnhost ready: false, restart count 0 Nov 11 21:01:59.078: INFO: csi-mockplugin-0 started at 2019-11-11 21:00:52 +0000 UTC (0+3 container statuses recorded) Nov 11 21:01:59.078: INFO: Container csi-provisioner ready: true, restart count 0 Nov 11 21:01:59.078: INFO: Container driver-registrar ready: true, restart count 0 Nov 11 21:01:59.078: INFO: Container mock ready: true, restart count 0 Nov 11 21:01:59.078: INFO: csi-mockplugin-attacher-0 started at 2019-11-11 21:00:52 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:59.078: INFO: Container csi-attacher ready: true, restart count 0 Nov 11 21:01:59.078: INFO: affinity-nodeport-rflj9 started at 2019-11-11 21:00:50 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:59.078: INFO: Container affinity-nodeport ready: true, restart count 0 Nov 11 21:01:59.078: INFO: affinity-nodeport-tvnm7 started at 2019-11-11 21:00:50 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:59.078: INFO: Container affinity-nodeport ready: true, restart count 0 Nov 11 21:01:59.078: INFO: csi-mockplugin-0 started at 2019-11-11 21:00:57 +0000 UTC (0+3 container statuses recorded) Nov 11 21:01:59.078: INFO: Container csi-provisioner ready: true, restart count 0 Nov 11 21:01:59.078: INFO: Container driver-registrar ready: true, restart count 0 Nov 11 21:01:59.078: INFO: Container mock ready: true, restart count 0 Nov 11 21:01:59.078: INFO: local-injector started at 2019-11-11 21:01:16 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:59.078: INFO: Container local-injector ready: true, restart count 0 Nov 11 21:01:59.078: INFO: pvc-volume-tester-zms7f started at 2019-11-11 21:01:24 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:59.078: INFO: Container volume-tester ready: false, restart count 0 W1111 21:01:59.136933 93347 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:01:59.301: INFO: Latency metrics for node k8s-agentpool-35271922-vmss000001 Nov 11 21:01:59.301: INFO: Logging node info for node k8s-master-35271922-vmss000000 Nov 11 21:01:59.358: INFO: Node Info: &Node{ObjectMeta:{k8s-master-35271922-vmss000000 /api/v1/nodes/k8s-master-35271922-vmss000000 f042d5b7-5ee8-4528-ae46-c997fef88c1c 27742 0 2019-11-11 20:48:07 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-35271922-vmss000000 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-35271922-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284363264 0} {<nil>} 7113636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6497931264 0} {<nil>} 6345636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:01:00 +0000 UTC,LastTransitionTime:2019-11-11 20:48:04 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:01:00 +0000 UTC,LastTransitionTime:2019-11-11 20:48:04 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:01:00 +0000 UTC,LastTransitionTime:2019-11-11 20:48:04 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:01:00 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.4,},NodeAddress{Type:Hostname,Address:k8s-master-35271922-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:8843d4f5cd0544d08ae501580db6a3a7,SystemUUID:41DCC090-746D-9E44-8888-8A77F0E0C51C,BootID:42d9eb02-ab49-4c2a-8791-7c5b2ac8f2bc,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:45e5b123e6188348715630acd2f7c1469c5b5181078ce90dc6f5399be4c17be1 k8sprow.azurecr.io/azure-cloud-controller-manager:1193984044413489154-9318bf1d],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:01:59.358: INFO: Logging kubelet events for node k8s-master-35271922-vmss000000 Nov 11 21:01:59.424: INFO: Logging pods the kubelet thinks is on node k8s-master-35271922-vmss000000 Nov 11 21:01:59.505: INFO: kube-addon-manager-k8s-master-35271922-vmss000000 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:59.505: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 11 21:01:59.505: INFO: kube-apiserver-k8s-master-35271922-vmss000000 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:59.505: INFO: Container kube-apiserver ready: true, restart count 0 Nov 11 21:01:59.505: INFO: kube-controller-manager-k8s-master-35271922-vmss000000 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:59.505: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 11 21:01:59.505: INFO: kube-scheduler-k8s-master-35271922-vmss000000 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:59.505: INFO: Container kube-scheduler ready: true, restart count 0 Nov 11 21:01:59.505: INFO: azure-ip-masq-agent-k7l5b started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:59.505: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:01:59.505: INFO: kube-proxy-8hzbt started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:59.505: INFO: Container kube-proxy ready: true, restart count 0 Nov 11 21:01:59.505: INFO: cloud-controller-manager-k8s-master-35271922-vmss000000 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:59.505: INFO: Container cloud-controller-manager ready: true, restart count 0 W1111 21:01:59.563285 93347 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:01:59.689: INFO: Latency metrics for node k8s-master-35271922-vmss000000 Nov 11 21:01:59.689: INFO: Logging node info for node k8s-master-35271922-vmss000001 Nov 11 21:01:59.744: INFO: Node Info: &Node{ObjectMeta:{k8s-master-35271922-vmss000001 /api/v1/nodes/k8s-master-35271922-vmss000001 0ff06487-2e4c-4c07-a6f2-d77d1a47bbcd 27784 0 2019-11-11 20:48:39 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-35271922-vmss000001 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.6.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-35271922-vmss/virtualMachines/1,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.6.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284363264 0} {<nil>} 7113636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6497931264 0} {<nil>} 6345636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:49:01 +0000 UTC,LastTransitionTime:2019-11-11 20:49:01 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:01:02 +0000 UTC,LastTransitionTime:2019-11-11 20:48:32 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:01:02 +0000 UTC,LastTransitionTime:2019-11-11 20:48:32 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:01:02 +0000 UTC,LastTransitionTime:2019-11-11 20:48:32 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:01:02 +0000 UTC,LastTransitionTime:2019-11-11 20:48:39 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.5,},NodeAddress{Type:Hostname,Address:k8s-master-35271922-vmss000001,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:24b17940193c48e38f59d10d619fb718,SystemUUID:25FDA250-5862-9148-98C8-17FFDD874AEC,BootID:cdd7987a-6ece-464d-a177-21f53ae0b68b,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:45e5b123e6188348715630acd2f7c1469c5b5181078ce90dc6f5399be4c17be1 k8sprow.azurecr.io/azure-cloud-controller-manager:1193984044413489154-9318bf1d],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:01:59.745: INFO: Logging kubelet events for node k8s-master-35271922-vmss000001 Nov 11 21:01:59.804: INFO: Logging pods the kubelet thinks is on node k8s-master-35271922-vmss000001 Nov 11 21:01:59.883: INFO: kube-apiserver-k8s-master-35271922-vmss000001 started at 2019-11-11 20:48:26 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:59.884: INFO: Container kube-apiserver ready: true, restart count 0 Nov 11 21:01:59.884: INFO: kube-controller-manager-k8s-master-35271922-vmss000001 started at 2019-11-11 20:48:26 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:59.884: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 11 21:01:59.884: INFO: kube-scheduler-k8s-master-35271922-vmss000001 started at 2019-11-11 20:48:26 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:59.884: INFO: Container kube-scheduler ready: true, restart count 0 Nov 11 21:01:59.884: INFO: azure-ip-masq-agent-bm2t5 started at 2019-11-11 20:48:40 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:59.884: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:01:59.884: INFO: kube-proxy-5fcm6 started at 2019-11-11 20:48:40 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:59.884: INFO: Container kube-proxy ready: true, restart count 0 Nov 11 21:01:59.884: INFO: cloud-controller-manager-k8s-master-35271922-vmss000001 started at 2019-11-11 20:48:26 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:59.884: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 11 21:01:59.884: INFO: kube-addon-manager-k8s-master-35271922-vmss000001 started at 2019-11-11 20:48:26 +0000 UTC (0+1 container statuses recorded) Nov 11 21:01:59.884: INFO: Container kube-addon-manager ready: true, restart count 0 W1111 21:01:59.950687 93347 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:02:00.080: INFO: Latency metrics for node k8s-master-35271922-vmss000001 Nov 11 21:02:00.081: INFO: Logging node info for node k8s-master-35271922-vmss000002 Nov 11 21:02:00.135: INFO: Node Info: &Node{ObjectMeta:{k8s-master-35271922-vmss000002 /api/v1/nodes/k8s-master-35271922-vmss000002 7fedbabf-dce6-48d1-994b-628369e7bcb5 27774 0 2019-11-11 20:48:08 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-35271922-vmss000002 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.4.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-35271922-vmss/virtualMachines/2,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284363264 0} {<nil>} 7113636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6497931264 0} {<nil>} 6345636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:01:01 +0000 UTC,LastTransitionTime:2019-11-11 20:48:03 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:01:01 +0000 UTC,LastTransitionTime:2019-11-11 20:48:03 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:01:01 +0000 UTC,LastTransitionTime:2019-11-11 20:48:03 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:01:01 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.6,},NodeAddress{Type:Hostname,Address:k8s-master-35271922-vmss000002,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:5ecd4c723b8845b18d85d341ba062204,SystemUUID:9E772AEA-D3D7-084D-98ED-1C1A511168B4,BootID:a73a4655-81c3-42e2-9d20-ef7e3cff3664,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:45e5b123e6188348715630acd2f7c1469c5b5181078ce90dc6f5399be4c17be1 k8sprow.azurecr.io/azure-cloud-controller-manager:1193984044413489154-9318bf1d],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:02:00.136: INFO: Logging kubelet events for node k8s-master-35271922-vmss000002 Nov 11 21:02:00.208: INFO: Logging pods the kubelet thinks is on node k8s-master-35271922-vmss000002 Nov 11 21:02:00.290: INFO: kube-apiserver-k8s-master-35271922-vmss000002 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:02:00.290: INFO: Container kube-apiserver ready: true, restart count 0 Nov 11 21:02:00.290: INFO: kube-controller-manager-k8s-master-35271922-vmss000002 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:02:00.290: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 11 21:02:00.290: INFO: kube-scheduler-k8s-master-35271922-vmss000002 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:02:00.290: INFO: Container kube-scheduler ready: true, restart count 0 Nov 11 21:02:00.290: INFO: azure-ip-masq-agent-cmbqm started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:02:00.290: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:02:00.290: INFO: kube-proxy-t2sp7 started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:02:00.290: INFO: Container kube-proxy ready: true, restart count 0 Nov 11 21:02:00.290: INFO: cloud-controller-manager-k8s-master-35271922-vmss000002 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:02:00.290: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 11 21:02:00.290: INFO: kube-addon-manager-k8s-master-35271922-vmss000002 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:02:00.290: INFO: Container kube-addon-manager ready: true, restart count 0 W1111 21:02:00.347362 93347 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:02:00.473: INFO: Latency metrics for node k8s-master-35271922-vmss000002 Nov 11 21:02:00.474: INFO: Logging node info for node k8s-master-35271922-vmss000003 Nov 11 21:02:00.545: INFO: Node Info: &Node{ObjectMeta:{k8s-master-35271922-vmss000003 /api/v1/nodes/k8s-master-35271922-vmss000003 d4d04c51-fc7b-4cdc-a1e3-69990a198568 27800 0 2019-11-11 20:48:10 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-35271922-vmss000003 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.5.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-35271922-vmss/virtualMachines/3,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.5.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284363264 0} {<nil>} 7113636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6497931264 0} {<nil>} 6345636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:01:02 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:01:02 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:01:02 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:01:02 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.7,},NodeAddress{Type:Hostname,Address:k8s-master-35271922-vmss000003,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:53117d11ac104f35951d19e4a4448f8f,SystemUUID:048B9870-F8AE-BC47-80D8-26703FED9760,BootID:fc64edd8-d9c5-4d6f-b9e7-c902146bc724,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:45e5b123e6188348715630acd2f7c1469c5b5181078ce90dc6f5399be4c17be1 k8sprow.azurecr.io/azure-cloud-controller-manager:1193984044413489154-9318bf1d],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:02:00.546: INFO: Logging kubelet events for node k8s-master-35271922-vmss000003 Nov 11 21:02:00.629: INFO: Logging pods the kubelet thinks is on node k8s-master-35271922-vmss000003 Nov 11 21:02:00.709: INFO: kube-apiserver-k8s-master-35271922-vmss000003 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:02:00.709: INFO: Container kube-apiserver ready: true, restart count 0 Nov 11 21:02:00.709: INFO: kube-controller-manager-k8s-master-35271922-vmss000003 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:02:00.709: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 11 21:02:00.709: INFO: kube-scheduler-k8s-master-35271922-vmss000003 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:02:00.709: INFO: Container kube-scheduler ready: true, restart count 0 Nov 11 21:02:00.709: INFO: azure-ip-masq-agent-rs668 started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:02:00.709: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:02:00.709: INFO: kube-proxy-tqjkf started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:02:00.709: INFO: Container kube-proxy ready: true, restart count 0 Nov 11 21:02:00.709: INFO: cloud-controller-manager-k8s-master-35271922-vmss000003 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:02:00.709: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 11 21:02:00.709: INFO: kube-addon-manager-k8s-master-35271922-vmss000003 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:02:00.709: INFO: Container kube-addon-manager ready: true, restart count 0 W1111 21:02:00.766657 93347 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:02:00.890: INFO: Latency metrics for node k8s-master-35271922-vmss000003 Nov 11 21:02:00.890: INFO: Logging node info for node k8s-master-35271922-vmss000004 Nov 11 21:02:00.955: INFO: Node Info: &Node{ObjectMeta:{k8s-master-35271922-vmss000004 /api/v1/nodes/k8s-master-35271922-vmss000004 edf29ad2-2fc8-4411-ac05-e1bf8781e340 27821 0 2019-11-11 20:48:11 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-35271922-vmss000004 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-35271922-vmss/virtualMachines/4,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284363264 0} {<nil>} 7113636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6497931264 0} {<nil>} 6345636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:01:03 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:01:03 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:01:03 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:01:03 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.8,},NodeAddress{Type:Hostname,Address:k8s-master-35271922-vmss000004,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:3ecc95ca67e14e2e87659efc6707e3b4,SystemUUID:0649BB2F-7C03-5743-A309-9BABFF3DD11A,BootID:1256b8d0-bf8e-4490-a45d-beaf41e45a09,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:45e5b123e6188348715630acd2f7c1469c5b5181078ce90dc6f5399be4c17be1 k8sprow.azurecr.io/azure-cloud-controller-manager:1193984044413489154-9318bf1d],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:02:00.956: INFO: Logging kubelet events for node k8s-master-35271922-vmss000004 Nov 11 21:02:01.015: INFO: Logging pods the kubelet thinks is on node k8s-master-35271922-vmss000004 Nov 11 21:02:01.095: INFO: kube-controller-manager-k8s-master-35271922-vmss000004 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:02:01.095: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 11 21:02:01.095: INFO: kube-scheduler-k8s-master-35271922-vmss000004 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:02:01.095: INFO: Container kube-scheduler ready: true, restart count 0 Nov 11 21:02:01.095: INFO: azure-ip-masq-agent-trzsm started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:02:01.095: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:02:01.095: INFO: kube-proxy-zrmqq started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:02:01.095: INFO: Container kube-proxy ready: true, restart count 0 Nov 11 21:02:01.095: INFO: cloud-controller-manager-k8s-master-35271922-vmss000004 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:02:01.095: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 11 21:02:01.095: INFO: kube-addon-manager-k8s-master-35271922-vmss000004 started at 2019-11-11 20:48:19 +0000 UTC (0+1 container statuses recorded) Nov 11 21:02:01.095: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 11 21:02:01.095: INFO: kube-apiserver-k8s-master-35271922-vmss000004 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:02:01.095: INFO: Container kube-apiserver ready: true, restart count 0 W1111 21:02:01.152589 93347 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:02:01.279: INFO: Latency metrics for node k8s-master-35271922-vmss000004 Nov 11 21:02:01.279: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "sched-preemption-path-420" for this suite. Nov 11 21:02:55.524: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered Nov 11 21:02:57.364: INFO: namespace sched-preemption-path-420 deletion completed in 56.028915932s [AfterEach] [sig-scheduling] PreemptionExecutionPath test/e2e/scheduling/preemption.go:274 Nov 11 21:02:57.420: INFO: List existing priorities: Nov 11 21:02:57.420: INFO: p1/1 created at 2019-11-11 21:00:00 +0000 UTC Nov 11 21:02:57.420: INFO: p2/2 created at 2019-11-11 21:00:00 +0000 UTC Nov 11 21:02:57.420: INFO: p3/3 created at 2019-11-11 21:00:01 +0000 UTC Nov 11 21:02:57.420: INFO: p4/4 created at 2019-11-11 21:00:01 +0000 UTC Nov 11 21:02:57.420: INFO: system-cluster-critical/2000000000 created at 2019-11-11 20:48:09 +0000 UTC Nov 11 21:02:57.420: INFO: system-node-critical/2000001000 created at 2019-11-11 20:48:09 +0000 UTC
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[sig\-scheduling\]\sPreemptionExecutionPath\sruns\sReplicaSets\sto\sverify\spreemption\srunning\spath$'
test/e2e/scheduling/preemption.go:345 Nov 11 21:04:45.153: Unexpected error: <*errors.errorString | 0xc00256c7e0>: { s: "replicaset \"rs-pod1\" never had desired number of .status.availableReplicas", } replicaset "rs-pod1" never had desired number of .status.availableReplicas occurred test/e2e/scheduling/preemption.go:510from junit_18.xml
[BeforeEach] [sig-scheduling] PreemptionExecutionPath test/e2e/framework/framework.go:151 �[1mSTEP�[0m: Creating a kubernetes client Nov 11 21:02:57.800: INFO: >>> kubeConfig: /workspace/aks147280421/kubeconfig/kubeconfig.westus2.json �[1mSTEP�[0m: Building a namespace api object, basename sched-preemption-path �[1mSTEP�[0m: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in sched-preemption-path-4167 �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace [BeforeEach] [sig-scheduling] PreemptionExecutionPath test/e2e/scheduling/preemption.go:302 �[1mSTEP�[0m: Finding an available node �[1mSTEP�[0m: Trying to launch a pod without a label to get a node which can launch it. �[1mSTEP�[0m: Explicitly delete pod here to free the resource it takes. Nov 11 21:03:44.556: INFO: found a healthy node: k8s-agentpool-35271922-vmss000000 [It] runs ReplicaSets to verify preemption running path test/e2e/scheduling/preemption.go:345 Nov 11 21:04:45.153: FAIL: Unexpected error: <*errors.errorString | 0xc00256c7e0>: { s: "replicaset \"rs-pod1\" never had desired number of .status.availableReplicas", } replicaset "rs-pod1" never had desired number of .status.availableReplicas occurred [AfterEach] [sig-scheduling] PreemptionExecutionPath test/e2e/framework/framework.go:152 �[1mSTEP�[0m: Collecting events from namespace "sched-preemption-path-4167". �[1mSTEP�[0m: Found 32 events. Nov 11 21:04:45.268: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-b4tmm: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 11 21:04:45.268: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-b4tmm: {default-scheduler } Scheduled: Successfully assigned sched-preemption-path-4167/rs-pod1-b4tmm to k8s-agentpool-35271922-vmss000000 Nov 11 21:04:45.268: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-b4tmm: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 11 21:04:45.268: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-cqshp: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 11 21:04:45.268: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-cqshp: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 11 21:04:45.268: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-cqshp: {default-scheduler } Scheduled: Successfully assigned sched-preemption-path-4167/rs-pod1-cqshp to k8s-agentpool-35271922-vmss000000 Nov 11 21:04:45.268: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-m6ffg: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 11 21:04:45.268: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-m6ffg: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 11 21:04:45.268: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-m6ffg: {default-scheduler } Scheduled: Successfully assigned sched-preemption-path-4167/rs-pod1-m6ffg to k8s-agentpool-35271922-vmss000000 Nov 11 21:04:45.268: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-vhnpc: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 11 21:04:45.268: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-vhnpc: {default-scheduler } Scheduled: Successfully assigned sched-preemption-path-4167/rs-pod1-vhnpc to k8s-agentpool-35271922-vmss000000 Nov 11 21:04:45.268: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-vhnpc: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 11 21:04:45.268: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-xn7lk: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 11 21:04:45.268: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-xn7lk: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 11 21:04:45.268: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-xn7lk: {default-scheduler } Scheduled: Successfully assigned sched-preemption-path-4167/rs-pod1-xn7lk to k8s-agentpool-35271922-vmss000000 Nov 11 21:04:45.268: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for without-label: {default-scheduler } Scheduled: Successfully assigned sched-preemption-path-4167/without-label to k8s-agentpool-35271922-vmss000000 Nov 11 21:04:45.268: INFO: At 2019-11-11 21:03:26 +0000 UTC - event for without-label: {kubelet k8s-agentpool-35271922-vmss000000} Pulling: Pulling image "k8s.gcr.io/pause:3.1" Nov 11 21:04:45.268: INFO: At 2019-11-11 21:03:27 +0000 UTC - event for without-label: {kubelet k8s-agentpool-35271922-vmss000000} Pulled: Successfully pulled image "k8s.gcr.io/pause:3.1" Nov 11 21:04:45.268: INFO: At 2019-11-11 21:03:32 +0000 UTC - event for without-label: {kubelet k8s-agentpool-35271922-vmss000000} Created: Created container without-label Nov 11 21:04:45.268: INFO: At 2019-11-11 21:03:40 +0000 UTC - event for without-label: {kubelet k8s-agentpool-35271922-vmss000000} Started: Started container without-label Nov 11 21:04:45.268: INFO: At 2019-11-11 21:03:44 +0000 UTC - event for rs-pod1: {replicaset-controller } SuccessfulCreate: Created pod: rs-pod1-m6ffg Nov 11 21:04:45.268: INFO: At 2019-11-11 21:03:44 +0000 UTC - event for without-label: {kubelet k8s-agentpool-35271922-vmss000000} Killing: Stopping container without-label Nov 11 21:04:45.268: INFO: At 2019-11-11 21:03:45 +0000 UTC - event for rs-pod1: {replicaset-controller } SuccessfulCreate: Created pod: rs-pod1-cqshp Nov 11 21:04:45.268: INFO: At 2019-11-11 21:03:45 +0000 UTC - event for rs-pod1: {replicaset-controller } SuccessfulCreate: Created pod: rs-pod1-xn7lk Nov 11 21:04:45.268: INFO: At 2019-11-11 21:03:45 +0000 UTC - event for rs-pod1: {replicaset-controller } SuccessfulCreate: Created pod: rs-pod1-vhnpc Nov 11 21:04:45.268: INFO: At 2019-11-11 21:03:45 +0000 UTC - event for rs-pod1: {replicaset-controller } SuccessfulCreate: Created pod: rs-pod1-b4tmm Nov 11 21:04:45.268: INFO: At 2019-11-11 21:04:28 +0000 UTC - event for rs-pod1-m6ffg: {kubelet k8s-agentpool-35271922-vmss000000} Pulling: Pulling image "k8s.gcr.io/pause:3.1" Nov 11 21:04:45.268: INFO: At 2019-11-11 21:04:29 +0000 UTC - event for rs-pod1-m6ffg: {kubelet k8s-agentpool-35271922-vmss000000} Pulled: Successfully pulled image "k8s.gcr.io/pause:3.1" Nov 11 21:04:45.268: INFO: At 2019-11-11 21:04:35 +0000 UTC - event for rs-pod1-m6ffg: {kubelet k8s-agentpool-35271922-vmss000000} Created: Created container pod1 Nov 11 21:04:45.268: INFO: At 2019-11-11 21:04:38 +0000 UTC - event for rs-pod1-m6ffg: {kubelet k8s-agentpool-35271922-vmss000000} Started: Started container pod1 Nov 11 21:04:45.268: INFO: At 2019-11-11 21:04:39 +0000 UTC - event for rs-pod1-cqshp: {kubelet k8s-agentpool-35271922-vmss000000} Pulling: Pulling image "k8s.gcr.io/pause:3.1" Nov 11 21:04:45.268: INFO: At 2019-11-11 21:04:40 +0000 UTC - event for rs-pod1-cqshp: {kubelet k8s-agentpool-35271922-vmss000000} Pulled: Successfully pulled image "k8s.gcr.io/pause:3.1" Nov 11 21:04:45.324: INFO: POD NODE PHASE GRACE CONDITIONS Nov 11 21:04:45.324: INFO: rs-pod1-b4tmm k8s-agentpool-35271922-vmss000000 Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:04:00 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:04:00 +0000 UTC ContainersNotReady containers with unready status: [pod1]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:04:00 +0000 UTC ContainersNotReady containers with unready status: [pod1]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:03:54 +0000 UTC }] Nov 11 21:04:45.324: INFO: rs-pod1-cqshp k8s-agentpool-35271922-vmss000000 Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:03:58 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:03:58 +0000 UTC ContainersNotReady containers with unready status: [pod1]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:03:58 +0000 UTC ContainersNotReady containers with unready status: [pod1]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:03:54 +0000 UTC }] Nov 11 21:04:45.324: INFO: rs-pod1-m6ffg k8s-agentpool-35271922-vmss000000 Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:03:56 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:03:56 +0000 UTC ContainersNotReady containers with unready status: [pod1]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:03:56 +0000 UTC ContainersNotReady containers with unready status: [pod1]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:03:54 +0000 UTC }] Nov 11 21:04:45.324: INFO: rs-pod1-vhnpc k8s-agentpool-35271922-vmss000000 Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:04:00 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:04:00 +0000 UTC ContainersNotReady containers with unready status: [pod1]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:04:00 +0000 UTC ContainersNotReady containers with unready status: [pod1]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:03:54 +0000 UTC }] Nov 11 21:04:45.324: INFO: rs-pod1-xn7lk k8s-agentpool-35271922-vmss000000 Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:03:57 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:03:57 +0000 UTC ContainersNotReady containers with unready status: [pod1]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:03:57 +0000 UTC ContainersNotReady containers with unready status: [pod1]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:03:54 +0000 UTC }] Nov 11 21:04:45.324: INFO: Nov 11 21:04:45.518: INFO: Logging node info for node k8s-agentpool-35271922-vmss000000 Nov 11 21:04:45.573: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool-35271922-vmss000000 /api/v1/nodes/k8s-agentpool-35271922-vmss000000 04da751e-f23a-470e-b29d-e591113d7271 31304 0 2019-11-11 20:48:07 +0000 UTC <nil> <nil> map[agentpool:agentpool beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool-35271922-vmss000000 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: storageprofile:managed storagetier:Premium_LRS] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-2030":"k8s-agentpool-35271922-vmss000000","csi-hostpath-ephemeral-7436":"k8s-agentpool-35271922-vmss000000","csi-hostpath-provisioning-3597":"k8s-agentpool-35271922-vmss000000","csi-hostpath-provisioning-9334":"k8s-agentpool-35271922-vmss000000","csi-hostpath-volume-5092":"k8s-agentpool-35271922-vmss000000","csi-hostpath-volume-expand-1449":"k8s-agentpool-35271922-vmss000000","csi-mock-csi-mock-volumes-1829":"csi-mock-csi-mock-volumes-1829","csi-mock-csi-mock-volumes-2481":"csi-mock-csi-mock-volumes-2481","csi-mock-csi-mock-volumes-9721":"csi-mock-csi-mock-volumes-9721"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-35271922-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},example.com/fakecpu: {{800 0} {<nil>} 800 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16797044736 0} {<nil>} 16403364Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},example.com/fakecpu: {{800 0} {<nil>} 800 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16010612736 0} {<nil>} 15635364Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:03:51 +0000 UTC,LastTransitionTime:2019-11-11 20:47:58 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:03:51 +0000 UTC,LastTransitionTime:2019-11-11 20:47:58 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:03:51 +0000 UTC,LastTransitionTime:2019-11-11 20:47:58 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:03:51 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.248.0.4,},NodeAddress{Type:Hostname,Address:k8s-agentpool-35271922-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:84c06f3ee41a47aaa558c81f34711895,SystemUUID:9BB24974-637B-EE4C-A221-0F74BB743424,BootID:59f7eb0c-c09c-4eae-a14b-0710f9e14004,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[quay.io/kubernetes_incubator/nfs-provisioner@sha256:df762117e3c891f2d2ddff46ecb0776ba1f9f3c44cfd7739b0683bcd7a7954a8 quay.io/kubernetes_incubator/nfs-provisioner:v2.2.2],SizeBytes:391772778,},ContainerImage{Names:[gluster/glusterdynamic-provisioner@sha256:90067cb05a7d217651e84576935934fd9dcff8e6bcdcbaa416bbf36fcd09dbd1 gluster/glusterdynamic-provisioner:v1.0],SizeBytes:373281573,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/volume/nfs@sha256:c2ad734346f608a5f7d69cfded93c4e8094069320657bd372d12ba21dea3ea71 gcr.io/kubernetes-e2e-test-images/volume/nfs:1.0],SizeBytes:225358913,},ContainerImage{Names:[httpd@sha256:addd70e4ee83f3bc9a4c1c7c41e37927ba47faf639312fc936df3afad7926f5a httpd:2.4.39-alpine],SizeBytes:126894770,},ContainerImage{Names:[httpd@sha256:eb8ccf084cf3e80eece1add239effefd171eb39adbc154d33c14260d905d4060 httpd:2.4.38-alpine],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1],SizeBytes:121711221,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727 gcr.io/kubernetes-e2e-test-images/agnhost:2.6],SizeBytes:57345321,},ContainerImage{Names:[quay.io/k8scsi/csi-provisioner@sha256:0efcb424f1dde9b9fb11a1a14f2e48ab47e1c3f08bc3a929990dcfcb1f7ab34f quay.io/k8scsi/csi-provisioner:v1.4.0-rc1],SizeBytes:54431016,},ContainerImage{Names:[quay.io/k8scsi/csi-snapshotter@sha256:e3d3e742e32d00488fdb401045b9b1d033d7ca0ab6e760f77b24750fc95e5f70 quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1],SizeBytes:51703561,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0],SizeBytes:46226754,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:0aba670b4d9d6b2e720bbf575d733156c676b693ca26501235444490300db838 quay.io/k8scsi/csi-attacher:v1.1.0],SizeBytes:42839085,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:7d46fb6eb8b890dc546029d1565d502b4a1d974d33625c6ee2bc7991b77fc1a1 quay.io/k8scsi/csi-resizer:v0.2.0],SizeBytes:42817100,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:f315c9042e56def3c05c6b04fe79ec9da6d39ddc557ca365a76cf35964ea08b6 quay.io/k8scsi/csi-resizer:v0.1.0],SizeBytes:42623056,},ContainerImage{Names:[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0],SizeBytes:42488424,},ContainerImage{Names:[quay.io/k8scsi/hostpathplugin@sha256:b4826e492fc1762fceaf9726f41575ca0a4567864d3d235da874818de18039de quay.io/k8scsi/hostpathplugin:v1.2.0-rc5],SizeBytes:28761497,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7],SizeBytes:22909487,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/echoserver@sha256:e9ba514b896cdf559eef8788b66c2c3ee55f3572df617647b4b0d8b6bf81cf19 gcr.io/kubernetes-e2e-test-images/echoserver:2.2],SizeBytes:21692741,},ContainerImage{Names:[quay.io/k8scsi/mock-driver@sha256:e0eed916b7d970bad2b7d9875f9ad16932f987f0f3d91ec5d86da68b0b5cc9d1 quay.io/k8scsi/mock-driver:v2.1.0],SizeBytes:16226335,},ContainerImage{Names:[nginx@sha256:485b610fefec7ff6c463ced9623314a04ed67e3945b9c08d7e53a47f6d108dc7 nginx:1.14-alpine],SizeBytes:16032814,},ContainerImage{Names:[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0],SizeBytes:15815995,},ContainerImage{Names:[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0],SizeBytes:14967303,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nautilus@sha256:33a732d4c42a266912a5091598a0f07653c9134db4b8d571690d8afd509e0bfc gcr.io/kubernetes-e2e-test-images/nautilus:1.0],SizeBytes:4753501,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 gcr.io/kubernetes-e2e-test-images/mounttest:1.0],SizeBytes:1563521,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29],SizeBytes:1154361,},ContainerImage{Names:[busybox@sha256:bbc3a03235220b170ba48a157dd097dd1379299370e1ed99ce976df0355d24f0 busybox:1.27],SizeBytes:1129289,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea k8s.gcr.io/pause-amd64:3.1 k8s.gcr.io/pause:3.1],SizeBytes:742472,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-9187^4],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-9187^4,DevicePath:,},},Config:nil,},} Nov 11 21:04:45.574: INFO: Logging kubelet events for node k8s-agentpool-35271922-vmss000000 Nov 11 21:04:45.633: INFO: Logging pods the kubelet thinks is on node k8s-agentpool-35271922-vmss000000 Nov 11 21:04:45.774: INFO: security-context-01f30ce1-6d56-487a-8b96-0d1ec40ea9b9 started at 2019-11-11 21:04:23 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:45.774: INFO: Container write-pod ready: false, restart count 0 Nov 11 21:04:45.774: INFO: csi-hostpathplugin-0 started at 2019-11-11 21:00:32 +0000 UTC (0+3 container statuses recorded) Nov 11 21:04:45.774: INFO: Container hostpath ready: false, restart count 0 Nov 11 21:04:45.774: INFO: Container liveness-probe ready: false, restart count 0 Nov 11 21:04:45.774: INFO: Container node-driver-registrar ready: false, restart count 0 Nov 11 21:04:45.774: INFO: ss2-0 started at 2019-11-11 21:01:16 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:45.774: INFO: Container webserver ready: true, restart count 0 Nov 11 21:04:45.774: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:03:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:45.774: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:04:45.774: INFO: azure-ip-masq-agent-c4mb2 started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:45.774: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:04:45.774: INFO: failure-1 started at 2019-11-11 21:03:50 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:45.774: INFO: Container failure-1 ready: false, restart count 0 Nov 11 21:04:45.774: INFO: csi-hostpath-resizer-0 started at 2019-11-11 21:00:11 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:45.774: INFO: Container csi-resizer ready: true, restart count 0 Nov 11 21:04:45.774: INFO: csi-snapshotter-0 started at 2019-11-11 21:02:41 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:45.774: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 11 21:04:45.774: INFO: netserver-1 started at 2019-11-11 21:03:45 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:45.774: INFO: Container webserver ready: false, restart count 0 Nov 11 21:04:45.774: INFO: pod-service-account-mountsa-mountspec started at 2019-11-11 21:04:44 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:45.774: INFO: Container token-test ready: false, restart count 0 Nov 11 21:04:45.774: INFO: keyvault-flexvolume-zzp7p started at 2019-11-11 20:48:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:45.774: INFO: Container keyvault-flexvolume ready: true, restart count 0 Nov 11 21:04:45.774: INFO: csi-snapshotter-0 started at 2019-11-11 21:00:11 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:45.774: INFO: Container csi-snapshotter ready: false, restart count 0 Nov 11 21:04:45.774: INFO: rs-pod1-cqshp started at 2019-11-11 21:03:58 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:45.774: INFO: Container pod1 ready: false, restart count 0 Nov 11 21:04:45.774: INFO: pvc-volume-tester-qw7lc started at 2019-11-11 21:02:33 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:45.774: INFO: Container volume-tester ready: true, restart count 0 Nov 11 21:04:45.774: INFO: csi-hostpath-resizer-0 started at 2019-11-11 21:02:40 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:45.774: INFO: Container csi-resizer ready: true, restart count 0 Nov 11 21:04:45.774: INFO: local-injector started at 2019-11-11 21:02:14 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:45.774: INFO: Container local-injector ready: false, restart count 0 Nov 11 21:04:45.774: INFO: rs-pod1-m6ffg started at 2019-11-11 21:03:56 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:45.774: INFO: Container pod1 ready: false, restart count 0 Nov 11 21:04:45.774: INFO: rs-pod1-vhnpc started at 2019-11-11 21:04:00 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:45.774: INFO: Container pod1 ready: false, restart count 0 Nov 11 21:04:45.775: INFO: csi-mockplugin-0 started at 2019-11-11 21:01:38 +0000 UTC (0+3 container statuses recorded) Nov 11 21:04:45.775: INFO: Container csi-provisioner ready: true, restart count 0 Nov 11 21:04:45.775: INFO: Container driver-registrar ready: true, restart count 0 Nov 11 21:04:45.775: INFO: Container mock ready: true, restart count 0 Nov 11 21:04:45.775: INFO: kubernetes-dashboard-65966766b9-lv2x8 started at 2019-11-11 20:48:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:45.775: INFO: Container kubernetes-dashboard ready: true, restart count 0 Nov 11 21:04:45.775: INFO: csi-mockplugin-attacher-0 started at 2019-11-11 21:01:39 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:45.775: INFO: Container csi-attacher ready: true, restart count 0 Nov 11 21:04:45.775: INFO: rs-pod1-xn7lk started at 2019-11-11 21:03:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:45.775: INFO: Container pod1 ready: false, restart count 0 Nov 11 21:04:45.775: INFO: csi-hostpath-provisioner-0 started at 2019-11-11 21:00:11 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:45.775: INFO: Container csi-provisioner ready: true, restart count 0 Nov 11 21:04:45.775: INFO: success started at 2019-11-11 21:02:58 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:45.775: INFO: Container success ready: false, restart count 0 Nov 11 21:04:45.775: INFO: csi-hostpathplugin-0 started at 2019-11-11 21:04:26 +0000 UTC (0+3 container statuses recorded) Nov 11 21:04:45.775: INFO: Container hostpath ready: false, restart count 0 Nov 11 21:04:45.775: INFO: Container liveness-probe ready: false, restart count 0 Nov 11 21:04:45.775: INFO: Container node-driver-registrar ready: false, restart count 0 Nov 11 21:04:45.775: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:02:16 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:45.775: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:04:45.775: INFO: pod-subpath-test-local-preprovisionedpv-xq6g started at 2019-11-11 21:03:05 +0000 UTC (2+1 container statuses recorded) Nov 11 21:04:45.775: INFO: Init container init-volume-local-preprovisionedpv-xq6g ready: true, restart count 0 Nov 11 21:04:45.775: INFO: Init container test-init-volume-local-preprovisionedpv-xq6g ready: true, restart count 0 Nov 11 21:04:45.775: INFO: Container test-container-subpath-local-preprovisionedpv-xq6g ready: false, restart count 0 Nov 11 21:04:45.775: INFO: rs-pod1-b4tmm started at 2019-11-11 21:04:00 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:45.775: INFO: Container pod1 ready: false, restart count 0 Nov 11 21:04:45.775: INFO: csi-hostpath-attacher-0 started at 2019-11-11 21:00:10 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:45.775: INFO: Container csi-attacher ready: false, restart count 0 Nov 11 21:04:45.775: INFO: csi-hostpath-provisioner-0 started at 2019-11-11 21:02:40 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:45.775: INFO: Container csi-provisioner ready: true, restart count 0 Nov 11 21:04:45.775: INFO: pod-1 started at 2019-11-11 21:04:35 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:45.775: INFO: Container busybox ready: false, restart count 0 Nov 11 21:04:45.775: INFO: blobfuse-flexvol-installer-dmnhw started at 2019-11-11 20:48:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:45.775: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 Nov 11 21:04:45.775: INFO: csi-hostpath-attacher-0 started at 2019-11-11 21:02:37 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:45.775: INFO: Container csi-attacher ready: true, restart count 0 Nov 11 21:04:45.775: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:01:24 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:45.775: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:04:45.775: INFO: coredns-87f5d796-6mz2l started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:45.775: INFO: Container coredns ready: true, restart count 0 Nov 11 21:04:45.775: INFO: pod-service-account-nomountsa started at 2019-11-11 21:04:41 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:45.775: INFO: Container token-test ready: false, restart count 0 Nov 11 21:04:45.775: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:03:06 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:45.775: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:04:45.775: INFO: pod-service-account-defaultsa-mountspec started at 2019-11-11 21:04:42 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:45.775: INFO: Container token-test ready: false, restart count 0 Nov 11 21:04:45.775: INFO: e2e-test-httpd-rc-vx27g started at 2019-11-11 21:04:05 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:45.775: INFO: Container e2e-test-httpd-rc ready: false, restart count 0 Nov 11 21:04:45.775: INFO: dns-test-a3c4ef04-b970-439e-a676-4641849423b3 started at <nil> (0+0 container statuses recorded) Nov 11 21:04:45.775: INFO: kube-proxy-nwdpz started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:45.775: INFO: Container kube-proxy ready: true, restart count 0 W1111 21:04:45.832195 93347 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:04:45.975: INFO: Latency metrics for node k8s-agentpool-35271922-vmss000000 Nov 11 21:04:45.975: INFO: Logging node info for node k8s-agentpool-35271922-vmss000001 Nov 11 21:04:46.030: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool-35271922-vmss000001 /api/v1/nodes/k8s-agentpool-35271922-vmss000001 af5ba0b6-4f7f-4468-9d3c-b3e116ca81ca 31871 0 2019-11-11 20:48:13 +0000 UTC <nil> <nil> map[agentpool:agentpool beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool-35271922-vmss000001 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: storageprofile:managed storagetier:Premium_LRS] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-provisioning-2466":"k8s-agentpool-35271922-vmss000001","csi-hostpath-provisioning-265":"k8s-agentpool-35271922-vmss000001","csi-hostpath-provisioning-310":"k8s-agentpool-35271922-vmss000001","csi-hostpath-volume-expand-3983":"k8s-agentpool-35271922-vmss000001","csi-mock-csi-mock-volumes-2781":"csi-mock-csi-mock-volumes-2781","csi-mock-csi-mock-volumes-2838":"csi-mock-csi-mock-volumes-2838","csi-mock-csi-mock-volumes-797":"csi-mock-csi-mock-volumes-797"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-35271922-vmss/virtualMachines/1,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16797044736 0} {<nil>} 16403364Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16010612736 0} {<nil>} 15635364Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:04:16 +0000 UTC,LastTransitionTime:2019-11-11 20:48:13 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:04:16 +0000 UTC,LastTransitionTime:2019-11-11 20:48:13 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:04:16 +0000 UTC,LastTransitionTime:2019-11-11 20:48:13 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:04:16 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.248.0.5,},NodeAddress{Type:Hostname,Address:k8s-agentpool-35271922-vmss000001,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:1df8c22200b84e64b76150d232ecb929,SystemUUID:B03EE801-7A2D-7140-BE20-B2C844575A73,BootID:d32206f7-48b0-46c5-a542-39721566b456,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[quay.io/kubernetes_incubator/nfs-provisioner@sha256:df762117e3c891f2d2ddff46ecb0776ba1f9f3c44cfd7739b0683bcd7a7954a8 quay.io/kubernetes_incubator/nfs-provisioner:v2.2.2],SizeBytes:391772778,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/volume/nfs@sha256:c2ad734346f608a5f7d69cfded93c4e8094069320657bd372d12ba21dea3ea71 gcr.io/kubernetes-e2e-test-images/volume/nfs:1.0],SizeBytes:225358913,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/jessie-dnsutils@sha256:ad583e33cb284f7ef046673809b146ec4053cda19b54a85d2b180a86169715eb gcr.io/kubernetes-e2e-test-images/jessie-dnsutils:1.0],SizeBytes:195659796,},ContainerImage{Names:[httpd@sha256:addd70e4ee83f3bc9a4c1c7c41e37927ba47faf639312fc936df3afad7926f5a httpd:2.4.39-alpine],SizeBytes:126894770,},ContainerImage{Names:[httpd@sha256:eb8ccf084cf3e80eece1add239effefd171eb39adbc154d33c14260d905d4060 httpd:2.4.38-alpine],SizeBytes:123781643,},ContainerImage{Names:[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0],SizeBytes:71821984,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727 gcr.io/kubernetes-e2e-test-images/agnhost:2.6],SizeBytes:57345321,},ContainerImage{Names:[quay.io/k8scsi/csi-provisioner@sha256:0efcb424f1dde9b9fb11a1a14f2e48ab47e1c3f08bc3a929990dcfcb1f7ab34f quay.io/k8scsi/csi-provisioner:v1.4.0-rc1],SizeBytes:54431016,},ContainerImage{Names:[quay.io/k8scsi/csi-snapshotter@sha256:e3d3e742e32d00488fdb401045b9b1d033d7ca0ab6e760f77b24750fc95e5f70 quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1],SizeBytes:51703561,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0],SizeBytes:46226754,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:0aba670b4d9d6b2e720bbf575d733156c676b693ca26501235444490300db838 quay.io/k8scsi/csi-attacher:v1.1.0],SizeBytes:42839085,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:7d46fb6eb8b890dc546029d1565d502b4a1d974d33625c6ee2bc7991b77fc1a1 quay.io/k8scsi/csi-resizer:v0.2.0],SizeBytes:42817100,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:f315c9042e56def3c05c6b04fe79ec9da6d39ddc557ca365a76cf35964ea08b6 quay.io/k8scsi/csi-resizer:v0.1.0],SizeBytes:42623056,},ContainerImage{Names:[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1],SizeBytes:42541759,},ContainerImage{Names:[gcr.io/google-containers/debian-base@sha256:6966a0aedd7592c18ff2dd803c08bd85780ee19f5e3a2e7cf908a4cd837afcde gcr.io/google-containers/debian-base:0.4.1],SizeBytes:42323657,},ContainerImage{Names:[redis@sha256:50899ea1ceed33fa03232f3ac57578a424faa1742c1ac9c7a7bdb95cdf19b858 redis:5.0.5-alpine],SizeBytes:29331594,},ContainerImage{Names:[quay.io/k8scsi/hostpathplugin@sha256:b4826e492fc1762fceaf9726f41575ca0a4567864d3d235da874818de18039de quay.io/k8scsi/hostpathplugin:v1.2.0-rc5],SizeBytes:28761497,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7],SizeBytes:22909487,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/echoserver@sha256:e9ba514b896cdf559eef8788b66c2c3ee55f3572df617647b4b0d8b6bf81cf19 gcr.io/kubernetes-e2e-test-images/echoserver:2.2],SizeBytes:21692741,},ContainerImage{Names:[quay.io/k8scsi/mock-driver@sha256:e0eed916b7d970bad2b7d9875f9ad16932f987f0f3d91ec5d86da68b0b5cc9d1 quay.io/k8scsi/mock-driver:v2.1.0],SizeBytes:16226335,},ContainerImage{Names:[nginx@sha256:485b610fefec7ff6c463ced9623314a04ed67e3945b9c08d7e53a47f6d108dc7 nginx:1.14-alpine],SizeBytes:16032814,},ContainerImage{Names:[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0],SizeBytes:15815995,},ContainerImage{Names:[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0],SizeBytes:14967303,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/dnsutils@sha256:2abeee84efb79c14d731966e034af33bf324d3b26ca28497555511ff094b3ddd gcr.io/kubernetes-e2e-test-images/dnsutils:1.1],SizeBytes:9349974,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nonewprivs@sha256:10066e9039219449fe3c81f38fe01928f87914150768ab81b62a468e51fa7411 gcr.io/kubernetes-e2e-test-images/nonewprivs:1.0],SizeBytes:6757579,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nautilus@sha256:33a732d4c42a266912a5091598a0f07653c9134db4b8d571690d8afd509e0bfc gcr.io/kubernetes-e2e-test-images/nautilus:1.0],SizeBytes:4753501,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/test-webserver@sha256:7f93d6e32798ff28bc6289254d0c2867fe2c849c8e46edc50f8624734309812e gcr.io/kubernetes-e2e-test-images/test-webserver:1.0],SizeBytes:4732240,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:4206620,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 gcr.io/kubernetes-e2e-test-images/mounttest:1.0],SizeBytes:1563521,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest-user@sha256:17319ca525ee003681fccf7e8c6b1b910ff4f49b653d939ac7f9b6e7c463933d gcr.io/kubernetes-e2e-test-images/mounttest-user:1.0],SizeBytes:1450451,},ContainerImage{Names:[busybox@sha256:1303dbf110c57f3edf68d9f5a16c082ec06c4cf7604831669faf2c712260b5a0 busybox:latest],SizeBytes:1219790,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29],SizeBytes:1154361,},ContainerImage{Names:[busybox@sha256:bbc3a03235220b170ba48a157dd097dd1379299370e1ed99ce976df0355d24f0 busybox:1.27],SizeBytes:1129289,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea k8s.gcr.io/pause-amd64:3.1 k8s.gcr.io/pause:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:04:46.031: INFO: Logging kubelet events for node k8s-agentpool-35271922-vmss000001 Nov 11 21:04:46.100: INFO: Logging pods the kubelet thinks is on node k8s-agentpool-35271922-vmss000001 Nov 11 21:04:46.187: INFO: external-provisioner-tr552 started at 2019-11-11 21:02:52 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:46.187: INFO: Container nfs-provisioner ready: true, restart count 0 Nov 11 21:04:46.187: INFO: pod-0 started at 2019-11-11 21:04:37 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:46.187: INFO: Container busybox ready: false, restart count 0 Nov 11 21:04:46.187: INFO: pod-service-account-defaultsa started at 2019-11-11 21:04:42 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:46.187: INFO: Container token-test ready: false, restart count 0 Nov 11 21:04:46.187: INFO: pod-service-account-nomountsa-mountspec started at 2019-11-11 21:04:44 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:46.187: INFO: Container token-test ready: false, restart count 0 Nov 11 21:04:46.187: INFO: pod-subpath-test-local-preprovisionedpv-jkjx started at 2019-11-11 21:02:45 +0000 UTC (2+1 container statuses recorded) Nov 11 21:04:46.187: INFO: Init container init-volume-local-preprovisionedpv-jkjx ready: true, restart count 0 Nov 11 21:04:46.187: INFO: Init container test-init-volume-local-preprovisionedpv-jkjx ready: false, restart count 0 Nov 11 21:04:46.187: INFO: Container test-container-subpath-local-preprovisionedpv-jkjx ready: false, restart count 0 Nov 11 21:04:46.187: INFO: pod-subpath-test-local-preprovisionedpv-gmbt started at 2019-11-11 21:02:49 +0000 UTC (2+1 container statuses recorded) Nov 11 21:04:46.187: INFO: Init container init-volume-local-preprovisionedpv-gmbt ready: true, restart count 0 Nov 11 21:04:46.187: INFO: Init container test-init-subpath-local-preprovisionedpv-gmbt ready: true, restart count 0 Nov 11 21:04:46.187: INFO: Container test-container-subpath-local-preprovisionedpv-gmbt ready: false, restart count 0 Nov 11 21:04:46.187: INFO: foo-zwzns started at 2019-11-11 21:04:40 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:46.187: INFO: Container c ready: false, restart count 0 Nov 11 21:04:46.187: INFO: pod-subpath-test-nfs-dynamicpv-df72 started at 2019-11-11 21:04:13 +0000 UTC (1+1 container statuses recorded) Nov 11 21:04:46.187: INFO: Init container init-volume-nfs-dynamicpv-df72 ready: false, restart count 0 Nov 11 21:04:46.187: INFO: Container test-container-subpath-nfs-dynamicpv-df72 ready: false, restart count 0 Nov 11 21:04:46.187: INFO: pod-service-account-defaultsa-nomountspec started at 2019-11-11 21:04:45 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:46.188: INFO: Container token-test ready: false, restart count 0 Nov 11 21:04:46.188: INFO: csi-hostpathplugin-0 started at 2019-11-11 21:03:26 +0000 UTC (0+3 container statuses recorded) Nov 11 21:04:46.188: INFO: Container hostpath ready: false, restart count 0 Nov 11 21:04:46.188: INFO: Container liveness-probe ready: false, restart count 0 Nov 11 21:04:46.188: INFO: Container node-driver-registrar ready: false, restart count 0 Nov 11 21:04:46.188: INFO: pod-721aa7fe-950e-4bbc-a01b-73d663f42721 started at 2019-11-11 21:04:15 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:46.188: INFO: Container test-container ready: false, restart count 0 Nov 11 21:04:46.188: INFO: keyvault-flexvolume-2s2wq started at 2019-11-11 20:48:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:46.188: INFO: Container keyvault-flexvolume ready: true, restart count 0 Nov 11 21:04:46.188: INFO: ss2-1 started at 2019-11-11 21:01:47 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:46.188: INFO: Container webserver ready: true, restart count 0 Nov 11 21:04:46.188: INFO: httpd started at 2019-11-11 21:01:56 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:46.188: INFO: Container httpd ready: true, restart count 0 Nov 11 21:04:46.188: INFO: foo-2z7cl started at 2019-11-11 21:04:41 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:46.188: INFO: Container c ready: false, restart count 0 Nov 11 21:04:46.188: INFO: csi-hostpath-attacher-0 started at 2019-11-11 21:03:25 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:46.188: INFO: Container csi-attacher ready: true, restart count 0 Nov 11 21:04:46.188: INFO: csi-hostpath-provisioner-0 started at 2019-11-11 21:03:28 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:46.188: INFO: Container csi-provisioner ready: false, restart count 0 Nov 11 21:04:46.188: INFO: pod-configmaps-706df129-298e-4358-b2c0-3118d1d6617c started at 2019-11-11 21:04:38 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:46.188: INFO: Container configmap-volume-test ready: false, restart count 0 Nov 11 21:04:46.188: INFO: kube-proxy-sgvk2 started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:46.188: INFO: Container kube-proxy ready: true, restart count 0 Nov 11 21:04:46.188: INFO: hostexec-k8s-agentpool-35271922-vmss000001 started at 2019-11-11 21:01:46 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:46.188: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:04:46.188: INFO: hostexec-k8s-agentpool-35271922-vmss000001 started at 2019-11-11 21:01:52 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:46.188: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:04:46.188: INFO: csi-mockplugin-attacher-0 started at 2019-11-11 21:00:52 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:46.188: INFO: Container csi-attacher ready: false, restart count 0 Nov 11 21:04:46.188: INFO: pod-service-account-mountsa started at 2019-11-11 21:04:43 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:46.188: INFO: Container token-test ready: false, restart count 0 Nov 11 21:04:46.188: INFO: csi-snapshotter-0 started at 2019-11-11 21:03:30 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:46.188: INFO: Container csi-snapshotter ready: false, restart count 0 Nov 11 21:04:46.188: INFO: ss2-2 started at 2019-11-11 21:02:28 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:46.188: INFO: Container webserver ready: false, restart count 0 Nov 11 21:04:46.188: INFO: csi-hostpath-resizer-0 started at 2019-11-11 21:03:29 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:46.188: INFO: Container csi-resizer ready: false, restart count 0 Nov 11 21:04:46.188: INFO: hostexec-k8s-agentpool-35271922-vmss000001 started at 2019-11-11 21:02:25 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:46.188: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:04:46.188: INFO: pod-db2b1b66-c55d-48d2-9d12-54ceaf21f1ab started at 2019-11-11 21:04:35 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:46.188: INFO: Container test-container ready: false, restart count 0 Nov 11 21:04:46.188: INFO: azure-ip-masq-agent-9427k started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:46.188: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:04:46.188: INFO: pod-subpath-test-local-preprovisionedpv-kfrm started at 2019-11-11 21:03:16 +0000 UTC (2+1 container statuses recorded) Nov 11 21:04:46.188: INFO: Init container init-volume-local-preprovisionedpv-kfrm ready: true, restart count 0 Nov 11 21:04:46.188: INFO: Init container test-init-volume-local-preprovisionedpv-kfrm ready: false, restart count 0 Nov 11 21:04:46.188: INFO: Container test-container-subpath-local-preprovisionedpv-kfrm ready: false, restart count 0 Nov 11 21:04:46.188: INFO: pod-service-account-mountsa-nomountspec started at <nil> (0+0 container statuses recorded) Nov 11 21:04:46.188: INFO: external-provisioner-nhsmb started at 2019-11-11 21:02:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:46.188: INFO: Container nfs-provisioner ready: true, restart count 0 Nov 11 21:04:46.188: INFO: pod-subpath-test-nfs-dynamicpv-glgw started at 2019-11-11 21:04:18 +0000 UTC (1+2 container statuses recorded) Nov 11 21:04:46.188: INFO: Init container test-init-subpath-nfs-dynamicpv-glgw ready: false, restart count 0 Nov 11 21:04:46.188: INFO: Container test-container-subpath-nfs-dynamicpv-glgw ready: false, restart count 0 Nov 11 21:04:46.188: INFO: Container test-container-volume-nfs-dynamicpv-glgw ready: false, restart count 0 Nov 11 21:04:46.188: INFO: inline-volume-tester-qlfvx started at 2019-11-11 21:03:30 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:46.188: INFO: Container csi-volume-tester ready: false, restart count 0 Nov 11 21:04:46.188: INFO: blobfuse-flexvol-installer-tvkwl started at 2019-11-11 20:48:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:46.188: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 Nov 11 21:04:46.188: INFO: tiller-deploy-7559b6b885-jsnn2 started at 2019-11-11 20:48:53 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:46.188: INFO: Container tiller ready: true, restart count 0 Nov 11 21:04:46.188: INFO: netserver-0 started at 2019-11-11 21:03:45 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:46.188: INFO: Container webserver ready: false, restart count 0 Nov 11 21:04:46.188: INFO: metrics-server-58ff8c5ddf-thxvp started at 2019-11-11 20:48:53 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:46.188: INFO: Container metrics-server ready: true, restart count 0 Nov 11 21:04:46.188: INFO: csi-mockplugin-0 started at 2019-11-11 21:00:52 +0000 UTC (0+3 container statuses recorded) Nov 11 21:04:46.188: INFO: Container csi-provisioner ready: false, restart count 0 Nov 11 21:04:46.188: INFO: Container driver-registrar ready: false, restart count 0 Nov 11 21:04:46.188: INFO: Container mock ready: false, restart count 0 Nov 11 21:04:46.188: INFO: pod-host-path-test started at 2019-11-11 21:04:24 +0000 UTC (0+2 container statuses recorded) Nov 11 21:04:46.188: INFO: Container test-container-1 ready: false, restart count 0 Nov 11 21:04:46.188: INFO: Container test-container-2 ready: false, restart count 0 W1111 21:04:46.244957 93347 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:04:55.967: INFO: Latency metrics for node k8s-agentpool-35271922-vmss000001 Nov 11 21:04:55.967: INFO: Logging node info for node k8s-master-35271922-vmss000000 Nov 11 21:04:56.022: INFO: Node Info: &Node{ObjectMeta:{k8s-master-35271922-vmss000000 /api/v1/nodes/k8s-master-35271922-vmss000000 f042d5b7-5ee8-4528-ae46-c997fef88c1c 31611 0 2019-11-11 20:48:07 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-35271922-vmss000000 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-35271922-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284363264 0} {<nil>} 7113636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6497931264 0} {<nil>} 6345636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:04:00 +0000 UTC,LastTransitionTime:2019-11-11 20:48:04 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:04:00 +0000 UTC,LastTransitionTime:2019-11-11 20:48:04 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:04:00 +0000 UTC,LastTransitionTime:2019-11-11 20:48:04 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:04:00 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.4,},NodeAddress{Type:Hostname,Address:k8s-master-35271922-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:8843d4f5cd0544d08ae501580db6a3a7,SystemUUID:41DCC090-746D-9E44-8888-8A77F0E0C51C,BootID:42d9eb02-ab49-4c2a-8791-7c5b2ac8f2bc,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:45e5b123e6188348715630acd2f7c1469c5b5181078ce90dc6f5399be4c17be1 k8sprow.azurecr.io/azure-cloud-controller-manager:1193984044413489154-9318bf1d],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:04:56.022: INFO: Logging kubelet events for node k8s-master-35271922-vmss000000 Nov 11 21:04:56.083: INFO: Logging pods the kubelet thinks is on node k8s-master-35271922-vmss000000 Nov 11 21:04:56.168: INFO: cloud-controller-manager-k8s-master-35271922-vmss000000 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:56.168: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 11 21:04:56.168: INFO: kube-addon-manager-k8s-master-35271922-vmss000000 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:56.168: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 11 21:04:56.168: INFO: kube-apiserver-k8s-master-35271922-vmss000000 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:56.168: INFO: Container kube-apiserver ready: true, restart count 0 Nov 11 21:04:56.168: INFO: kube-controller-manager-k8s-master-35271922-vmss000000 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:56.168: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 11 21:04:56.168: INFO: kube-scheduler-k8s-master-35271922-vmss000000 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:56.168: INFO: Container kube-scheduler ready: true, restart count 0 Nov 11 21:04:56.168: INFO: azure-ip-masq-agent-k7l5b started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:56.168: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:04:56.168: INFO: kube-proxy-8hzbt started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:56.168: INFO: Container kube-proxy ready: true, restart count 0 W1111 21:04:56.225137 93347 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:04:56.373: INFO: Latency metrics for node k8s-master-35271922-vmss000000 Nov 11 21:04:56.373: INFO: Logging node info for node k8s-master-35271922-vmss000001 Nov 11 21:04:56.429: INFO: Node Info: &Node{ObjectMeta:{k8s-master-35271922-vmss000001 /api/v1/nodes/k8s-master-35271922-vmss000001 0ff06487-2e4c-4c07-a6f2-d77d1a47bbcd 31631 0 2019-11-11 20:48:39 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-35271922-vmss000001 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.6.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-35271922-vmss/virtualMachines/1,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.6.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284363264 0} {<nil>} 7113636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6497931264 0} {<nil>} 6345636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:49:01 +0000 UTC,LastTransitionTime:2019-11-11 20:49:01 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:04:02 +0000 UTC,LastTransitionTime:2019-11-11 20:48:32 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:04:02 +0000 UTC,LastTransitionTime:2019-11-11 20:48:32 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:04:02 +0000 UTC,LastTransitionTime:2019-11-11 20:48:32 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:04:02 +0000 UTC,LastTransitionTime:2019-11-11 20:48:39 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.5,},NodeAddress{Type:Hostname,Address:k8s-master-35271922-vmss000001,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:24b17940193c48e38f59d10d619fb718,SystemUUID:25FDA250-5862-9148-98C8-17FFDD874AEC,BootID:cdd7987a-6ece-464d-a177-21f53ae0b68b,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:45e5b123e6188348715630acd2f7c1469c5b5181078ce90dc6f5399be4c17be1 k8sprow.azurecr.io/azure-cloud-controller-manager:1193984044413489154-9318bf1d],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:04:56.429: INFO: Logging kubelet events for node k8s-master-35271922-vmss000001 Nov 11 21:04:56.495: INFO: Logging pods the kubelet thinks is on node k8s-master-35271922-vmss000001 Nov 11 21:04:56.579: INFO: kube-addon-manager-k8s-master-35271922-vmss000001 started at 2019-11-11 20:48:26 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:56.579: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 11 21:04:56.579: INFO: kube-apiserver-k8s-master-35271922-vmss000001 started at 2019-11-11 20:48:26 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:56.579: INFO: Container kube-apiserver ready: true, restart count 0 Nov 11 21:04:56.579: INFO: kube-controller-manager-k8s-master-35271922-vmss000001 started at 2019-11-11 20:48:26 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:56.579: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 11 21:04:56.579: INFO: kube-scheduler-k8s-master-35271922-vmss000001 started at 2019-11-11 20:48:26 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:56.579: INFO: Container kube-scheduler ready: true, restart count 0 Nov 11 21:04:56.579: INFO: azure-ip-masq-agent-bm2t5 started at 2019-11-11 20:48:40 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:56.579: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:04:56.579: INFO: kube-proxy-5fcm6 started at 2019-11-11 20:48:40 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:56.579: INFO: Container kube-proxy ready: true, restart count 0 Nov 11 21:04:56.579: INFO: cloud-controller-manager-k8s-master-35271922-vmss000001 started at 2019-11-11 20:48:26 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:56.579: INFO: Container cloud-controller-manager ready: true, restart count 0 W1111 21:04:56.637193 93347 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:04:56.805: INFO: Latency metrics for node k8s-master-35271922-vmss000001 Nov 11 21:04:56.805: INFO: Logging node info for node k8s-master-35271922-vmss000002 Nov 11 21:04:56.896: INFO: Node Info: &Node{ObjectMeta:{k8s-master-35271922-vmss000002 /api/v1/nodes/k8s-master-35271922-vmss000002 7fedbabf-dce6-48d1-994b-628369e7bcb5 31623 0 2019-11-11 20:48:08 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-35271922-vmss000002 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.4.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-35271922-vmss/virtualMachines/2,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284363264 0} {<nil>} 7113636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6497931264 0} {<nil>} 6345636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:04:02 +0000 UTC,LastTransitionTime:2019-11-11 20:48:03 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:04:02 +0000 UTC,LastTransitionTime:2019-11-11 20:48:03 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:04:02 +0000 UTC,LastTransitionTime:2019-11-11 20:48:03 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:04:02 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.6,},NodeAddress{Type:Hostname,Address:k8s-master-35271922-vmss000002,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:5ecd4c723b8845b18d85d341ba062204,SystemUUID:9E772AEA-D3D7-084D-98ED-1C1A511168B4,BootID:a73a4655-81c3-42e2-9d20-ef7e3cff3664,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:45e5b123e6188348715630acd2f7c1469c5b5181078ce90dc6f5399be4c17be1 k8sprow.azurecr.io/azure-cloud-controller-manager:1193984044413489154-9318bf1d],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:04:56.896: INFO: Logging kubelet events for node k8s-master-35271922-vmss000002 Nov 11 21:04:56.968: INFO: Logging pods the kubelet thinks is on node k8s-master-35271922-vmss000002 Nov 11 21:04:57.053: INFO: azure-ip-masq-agent-cmbqm started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:57.053: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:04:57.053: INFO: kube-proxy-t2sp7 started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:57.053: INFO: Container kube-proxy ready: true, restart count 0 Nov 11 21:04:57.053: INFO: cloud-controller-manager-k8s-master-35271922-vmss000002 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:57.053: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 11 21:04:57.053: INFO: kube-addon-manager-k8s-master-35271922-vmss000002 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:57.054: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 11 21:04:57.054: INFO: kube-apiserver-k8s-master-35271922-vmss000002 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:57.054: INFO: Container kube-apiserver ready: true, restart count 0 Nov 11 21:04:57.054: INFO: kube-controller-manager-k8s-master-35271922-vmss000002 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:57.054: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 11 21:04:57.054: INFO: kube-scheduler-k8s-master-35271922-vmss000002 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:57.054: INFO: Container kube-scheduler ready: true, restart count 0 W1111 21:04:57.111514 93347 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:04:57.246: INFO: Latency metrics for node k8s-master-35271922-vmss000002 Nov 11 21:04:57.246: INFO: Logging node info for node k8s-master-35271922-vmss000003 Nov 11 21:04:57.303: INFO: Node Info: &Node{ObjectMeta:{k8s-master-35271922-vmss000003 /api/v1/nodes/k8s-master-35271922-vmss000003 d4d04c51-fc7b-4cdc-a1e3-69990a198568 31645 0 2019-11-11 20:48:10 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-35271922-vmss000003 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.5.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-35271922-vmss/virtualMachines/3,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.5.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284363264 0} {<nil>} 7113636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6497931264 0} {<nil>} 6345636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:04:03 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:04:03 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:04:03 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:04:03 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.7,},NodeAddress{Type:Hostname,Address:k8s-master-35271922-vmss000003,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:53117d11ac104f35951d19e4a4448f8f,SystemUUID:048B9870-F8AE-BC47-80D8-26703FED9760,BootID:fc64edd8-d9c5-4d6f-b9e7-c902146bc724,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:45e5b123e6188348715630acd2f7c1469c5b5181078ce90dc6f5399be4c17be1 k8sprow.azurecr.io/azure-cloud-controller-manager:1193984044413489154-9318bf1d],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:04:57.303: INFO: Logging kubelet events for node k8s-master-35271922-vmss000003 Nov 11 21:04:57.366: INFO: Logging pods the kubelet thinks is on node k8s-master-35271922-vmss000003 Nov 11 21:04:57.446: INFO: kube-apiserver-k8s-master-35271922-vmss000003 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:57.446: INFO: Container kube-apiserver ready: true, restart count 0 Nov 11 21:04:57.446: INFO: kube-controller-manager-k8s-master-35271922-vmss000003 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:57.446: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 11 21:04:57.446: INFO: kube-scheduler-k8s-master-35271922-vmss000003 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:57.446: INFO: Container kube-scheduler ready: true, restart count 0 Nov 11 21:04:57.446: INFO: azure-ip-masq-agent-rs668 started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:57.446: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:04:57.446: INFO: kube-proxy-tqjkf started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:57.446: INFO: Container kube-proxy ready: true, restart count 0 Nov 11 21:04:57.446: INFO: cloud-controller-manager-k8s-master-35271922-vmss000003 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:57.446: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 11 21:04:57.446: INFO: kube-addon-manager-k8s-master-35271922-vmss000003 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:57.446: INFO: Container kube-addon-manager ready: true, restart count 0 W1111 21:04:57.503675 93347 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:04:57.633: INFO: Latency metrics for node k8s-master-35271922-vmss000003 Nov 11 21:04:57.633: INFO: Logging node info for node k8s-master-35271922-vmss000004 Nov 11 21:04:57.699: INFO: Node Info: &Node{ObjectMeta:{k8s-master-35271922-vmss000004 /api/v1/nodes/k8s-master-35271922-vmss000004 edf29ad2-2fc8-4411-ac05-e1bf8781e340 31650 0 2019-11-11 20:48:11 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-35271922-vmss000004 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-35271922-vmss/virtualMachines/4,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284363264 0} {<nil>} 7113636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6497931264 0} {<nil>} 6345636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:04:04 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:04:04 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:04:04 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:04:04 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.8,},NodeAddress{Type:Hostname,Address:k8s-master-35271922-vmss000004,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:3ecc95ca67e14e2e87659efc6707e3b4,SystemUUID:0649BB2F-7C03-5743-A309-9BABFF3DD11A,BootID:1256b8d0-bf8e-4490-a45d-beaf41e45a09,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:45e5b123e6188348715630acd2f7c1469c5b5181078ce90dc6f5399be4c17be1 k8sprow.azurecr.io/azure-cloud-controller-manager:1193984044413489154-9318bf1d],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:04:57.700: INFO: Logging kubelet events for node k8s-master-35271922-vmss000004 Nov 11 21:04:57.792: INFO: Logging pods the kubelet thinks is on node k8s-master-35271922-vmss000004 Nov 11 21:04:57.873: INFO: kube-proxy-zrmqq started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:57.873: INFO: Container kube-proxy ready: true, restart count 0 Nov 11 21:04:57.873: INFO: cloud-controller-manager-k8s-master-35271922-vmss000004 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:57.873: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 11 21:04:57.873: INFO: kube-addon-manager-k8s-master-35271922-vmss000004 started at 2019-11-11 20:48:19 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:57.873: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 11 21:04:57.873: INFO: kube-apiserver-k8s-master-35271922-vmss000004 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:57.873: INFO: Container kube-apiserver ready: true, restart count 0 Nov 11 21:04:57.873: INFO: kube-controller-manager-k8s-master-35271922-vmss000004 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:57.873: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 11 21:04:57.873: INFO: kube-scheduler-k8s-master-35271922-vmss000004 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:57.873: INFO: Container kube-scheduler ready: true, restart count 0 Nov 11 21:04:57.873: INFO: azure-ip-masq-agent-trzsm started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:04:57.873: INFO: Container azure-ip-masq-agent ready: true, restart count 0 W1111 21:04:57.943710 93347 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:04:58.075: INFO: Latency metrics for node k8s-master-35271922-vmss000004 Nov 11 21:04:58.075: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "sched-preemption-path-4167" for this suite. Nov 11 21:07:52.365: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered Nov 11 21:07:54.226: INFO: namespace sched-preemption-path-4167 deletion completed in 2m56.040306682s [AfterEach] [sig-scheduling] PreemptionExecutionPath test/e2e/scheduling/preemption.go:274 Nov 11 21:07:54.288: INFO: List existing priorities: Nov 11 21:07:54.288: INFO: p1/1 created at 2019-11-11 21:03:44 +0000 UTC Nov 11 21:07:54.288: INFO: p2/2 created at 2019-11-11 21:03:44 +0000 UTC Nov 11 21:07:54.288: INFO: p3/3 created at 2019-11-11 21:03:44 +0000 UTC Nov 11 21:07:54.288: INFO: p4/4 created at 2019-11-11 21:03:44 +0000 UTC Nov 11 21:07:54.288: INFO: system-cluster-critical/2000000000 created at 2019-11-11 20:48:09 +0000 UTC Nov 11 21:07:54.288: INFO: system-node-critical/2000001000 created at 2019-11-11 20:48:09 +0000 UTC
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[sig\-storage\]\sCSI\sVolumes\s\[Driver\:\scsi\-hostpath\]\s\[Testpattern\:\sDynamic\sPV\s\(block\svolmode\)\]\svolume\-expand\sshould\snot\sallow\sexpansion\sof\spvcs\swithout\sAllowVolumeExpansion\sproperty$'
test/e2e/storage/testsuites/volume_expand.go:139 Nov 11 21:20:53.897: Unexpected error: <*errors.errorString | 0xc000536940>: { s: "PersistentVolumeClaims [csi-hostpaths9v8h] not all in phase Bound within 5m0s", } PersistentVolumeClaims [csi-hostpaths9v8h] not all in phase Bound within 5m0s occurred test/e2e/storage/testsuites/base.go:366from junit_03.xml
[BeforeEach] [Testpattern: Dynamic PV (block volmode)] volume-expand test/e2e/storage/testsuites/base.go:93 [BeforeEach] [Testpattern: Dynamic PV (block volmode)] volume-expand test/e2e/storage/testsuites/volume_expand.go:85 [BeforeEach] [Testpattern: Dynamic PV (block volmode)] volume-expand test/e2e/framework/framework.go:151 �[1mSTEP�[0m: Creating a kubernetes client Nov 11 21:15:51.304: INFO: >>> kubeConfig: /workspace/aks147280421/kubeconfig/kubeconfig.westus2.json �[1mSTEP�[0m: Building a namespace api object, basename volume-expand �[1mSTEP�[0m: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in volume-expand-5832 �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace [It] should not allow expansion of pvcs without AllowVolumeExpansion property test/e2e/storage/testsuites/volume_expand.go:139 �[1mSTEP�[0m: deploying csi-hostpath driver Nov 11 21:15:51.892: INFO: creating *v1.ServiceAccount: volume-expand-5832/csi-attacher Nov 11 21:15:51.948: INFO: creating *v1.ClusterRole: external-attacher-runner-volume-expand-5832 Nov 11 21:15:51.948: INFO: Define cluster role external-attacher-runner-volume-expand-5832 Nov 11 21:15:52.006: INFO: creating *v1.ClusterRoleBinding: csi-attacher-role-volume-expand-5832 Nov 11 21:15:52.061: INFO: creating *v1.Role: volume-expand-5832/external-attacher-cfg-volume-expand-5832 Nov 11 21:15:52.119: INFO: creating *v1.RoleBinding: volume-expand-5832/csi-attacher-role-cfg Nov 11 21:15:52.177: INFO: creating *v1.ServiceAccount: volume-expand-5832/csi-provisioner Nov 11 21:15:52.235: INFO: creating *v1.ClusterRole: external-provisioner-runner-volume-expand-5832 Nov 11 21:15:52.235: INFO: Define cluster role external-provisioner-runner-volume-expand-5832 Nov 11 21:15:52.292: INFO: creating *v1.ClusterRoleBinding: csi-provisioner-role-volume-expand-5832 Nov 11 21:15:52.350: INFO: creating *v1.Role: volume-expand-5832/external-provisioner-cfg-volume-expand-5832 Nov 11 21:15:52.406: INFO: creating *v1.RoleBinding: volume-expand-5832/csi-provisioner-role-cfg Nov 11 21:15:52.462: INFO: creating *v1.ServiceAccount: volume-expand-5832/csi-snapshotter Nov 11 21:15:52.517: INFO: creating *v1.ClusterRole: external-snapshotter-runner-volume-expand-5832 Nov 11 21:15:52.517: INFO: Define cluster role external-snapshotter-runner-volume-expand-5832 Nov 11 21:15:52.577: INFO: creating *v1.ClusterRoleBinding: csi-snapshotter-role-volume-expand-5832 Nov 11 21:15:52.632: INFO: creating *v1.Role: volume-expand-5832/external-snapshotter-leaderelection-volume-expand-5832 Nov 11 21:15:52.687: INFO: creating *v1.RoleBinding: volume-expand-5832/external-snapshotter-leaderelection Nov 11 21:15:52.747: INFO: creating *v1.ServiceAccount: volume-expand-5832/csi-resizer Nov 11 21:15:52.802: INFO: creating *v1.ClusterRole: external-resizer-runner-volume-expand-5832 Nov 11 21:15:52.802: INFO: Define cluster role external-resizer-runner-volume-expand-5832 Nov 11 21:15:52.857: INFO: creating *v1.ClusterRoleBinding: csi-resizer-role-volume-expand-5832 Nov 11 21:15:52.913: INFO: creating *v1.Role: volume-expand-5832/external-resizer-cfg-volume-expand-5832 Nov 11 21:15:52.973: INFO: creating *v1.RoleBinding: volume-expand-5832/csi-resizer-role-cfg Nov 11 21:15:53.028: INFO: creating *v1.Service: volume-expand-5832/csi-hostpath-attacher Nov 11 21:15:53.096: INFO: creating *v1.StatefulSet: volume-expand-5832/csi-hostpath-attacher Nov 11 21:15:53.153: INFO: creating *v1beta1.CSIDriver: csi-hostpath-volume-expand-5832 Nov 11 21:15:53.207: INFO: creating *v1.Service: volume-expand-5832/csi-hostpathplugin Nov 11 21:15:53.275: INFO: creating *v1.StatefulSet: volume-expand-5832/csi-hostpathplugin Nov 11 21:15:53.336: INFO: creating *v1.Service: volume-expand-5832/csi-hostpath-provisioner Nov 11 21:15:53.406: INFO: creating *v1.StatefulSet: volume-expand-5832/csi-hostpath-provisioner Nov 11 21:15:53.465: INFO: creating *v1.Service: volume-expand-5832/csi-hostpath-resizer Nov 11 21:15:53.537: INFO: creating *v1.StatefulSet: volume-expand-5832/csi-hostpath-resizer Nov 11 21:15:53.598: INFO: creating *v1.Service: volume-expand-5832/csi-snapshotter Nov 11 21:15:53.670: INFO: creating *v1.StatefulSet: volume-expand-5832/csi-snapshotter Nov 11 21:15:53.726: INFO: creating *v1.ClusterRoleBinding: psp-csi-hostpath-role-volume-expand-5832 Nov 11 21:15:53.781: INFO: Test running for native CSI Driver, not checking metrics Nov 11 21:15:53.781: INFO: Creating resource for dynamic PV �[1mSTEP�[0m: creating a StorageClass volume-expand-5832-csi-hostpath-volume-expand-5832-scb94cv �[1mSTEP�[0m: creating a claim Nov 11 21:15:53.896: INFO: Waiting up to 5m0s for PersistentVolumeClaims [csi-hostpaths9v8h] to have phase Bound Nov 11 21:15:53.948: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:15:56.002: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:15:58.079: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:16:00.135: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:16:02.188: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:16:04.243: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:16:06.295: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:16:08.349: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:16:10.405: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:16:12.458: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:16:14.510: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:16:16.563: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:16:18.616: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:16:20.670: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:16:22.722: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:16:24.775: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:16:26.827: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:16:28.879: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:16:30.931: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:16:32.984: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:16:35.035: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:16:37.088: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:16:39.143: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:16:41.196: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:16:43.249: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:16:45.303: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:16:47.362: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:16:49.414: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:16:51.467: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:16:53.520: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:16:55.576: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:16:57.629: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:16:59.684: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:17:01.743: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:17:03.796: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:17:05.851: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:17:07.904: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:17:09.956: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:17:12.009: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:17:14.124: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:17:16.176: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:17:18.235: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:17:20.294: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:17:22.351: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:17:24.403: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:17:26.463: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:17:28.516: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:17:30.568: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:17:32.620: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:17:34.673: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:17:36.728: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:17:38.780: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:17:40.833: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:17:42.885: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:17:44.937: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:17:46.991: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:17:49.043: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:17:51.096: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:17:53.148: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:17:55.200: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:17:57.259: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:17:59.328: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:18:01.384: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:18:03.442: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:18:05.495: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:18:07.547: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:18:09.600: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:18:11.653: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:18:13.708: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:18:15.760: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:18:17.813: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:18:19.866: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:18:21.919: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:18:23.971: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:18:26.024: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:18:28.077: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:18:30.146: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:18:32.197: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:18:34.250: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:18:36.302: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:18:38.355: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:18:40.410: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:18:42.462: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:18:44.514: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:18:46.567: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:18:48.620: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:18:50.672: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:18:52.726: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:18:54.779: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:18:56.832: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:18:58.884: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:19:00.938: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:19:02.991: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:19:05.044: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:19:07.096: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:19:09.150: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:19:11.208: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:19:13.264: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:19:15.318: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:19:17.373: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:19:19.434: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:19:21.486: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:19:23.539: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:19:25.592: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:19:27.647: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:19:29.699: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:19:31.751: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:19:33.804: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:19:35.856: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:19:37.909: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:19:39.962: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:19:42.014: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:19:44.067: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:19:46.153: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:19:48.205: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:19:50.258: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:19:52.310: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:19:54.363: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:19:56.420: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:19:58.473: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:20:00.530: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:20:02.582: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:20:04.634: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:20:06.686: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:20:08.739: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:20:10.790: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:20:12.843: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:20:14.896: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:20:16.949: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:20:19.034: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:20:21.094: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:20:23.147: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:20:25.199: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:20:27.251: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:20:29.304: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:20:31.356: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:20:33.411: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:20:35.465: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:20:37.522: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:20:39.575: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:20:41.627: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:20:43.678: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:20:45.737: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:20:47.791: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:20:49.843: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:20:51.895: INFO: PersistentVolumeClaim csi-hostpaths9v8h found but phase is Pending instead of Bound. Nov 11 21:20:53.897: FAIL: Unexpected error: <*errors.errorString | 0xc000536940>: { s: "PersistentVolumeClaims [csi-hostpaths9v8h] not all in phase Bound within 5m0s", } PersistentVolumeClaims [csi-hostpaths9v8h] not all in phase Bound within 5m0s occurred [AfterEach] [Testpattern: Dynamic PV (block volmode)] volume-expand test/e2e/framework/framework.go:152 �[1mSTEP�[0m: Collecting events from namespace "volume-expand-5832". �[1mSTEP�[0m: Found 66 events. Nov 11 21:20:53.999: INFO: At 2019-11-11 21:15:53 +0000 UTC - event for csi-hostpath-attacher: {statefulset-controller } SuccessfulCreate: create Pod csi-hostpath-attacher-0 in StatefulSet csi-hostpath-attacher successful Nov 11 21:20:53.999: INFO: At 2019-11-11 21:15:53 +0000 UTC - event for csi-hostpath-provisioner: {statefulset-controller } SuccessfulCreate: create Pod csi-hostpath-provisioner-0 in StatefulSet csi-hostpath-provisioner successful Nov 11 21:20:53.999: INFO: At 2019-11-11 21:15:53 +0000 UTC - event for csi-hostpath-resizer: {statefulset-controller } SuccessfulCreate: create Pod csi-hostpath-resizer-0 in StatefulSet csi-hostpath-resizer successful Nov 11 21:20:53.999: INFO: At 2019-11-11 21:15:53 +0000 UTC - event for csi-hostpathplugin: {statefulset-controller } SuccessfulCreate: create Pod csi-hostpathplugin-0 in StatefulSet csi-hostpathplugin successful Nov 11 21:20:53.999: INFO: At 2019-11-11 21:15:53 +0000 UTC - event for csi-hostpaths9v8h: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "csi-hostpath-volume-expand-5832" or manually created by system administrator Nov 11 21:20:53.999: INFO: At 2019-11-11 21:15:53 +0000 UTC - event for csi-snapshotter: {statefulset-controller } SuccessfulCreate: create Pod csi-snapshotter-0 in StatefulSet csi-snapshotter successful Nov 11 21:20:53.999: INFO: At 2019-11-11 21:16:12 +0000 UTC - event for csi-hostpathplugin: {statefulset-controller } FailedCreate: create Pod csi-hostpathplugin-0 in StatefulSet csi-hostpathplugin failed error: The POST operation against Pod could not be completed at this time, please try again. Nov 11 21:20:53.999: INFO: At 2019-11-11 21:16:12 +0000 UTC - event for csi-hostpathplugin: {statefulset-controller } RecreatingFailedPod: StatefulSet volume-expand-5832/csi-hostpathplugin is recreating failed Pod csi-hostpathplugin-0 Nov 11 21:20:54.000: INFO: At 2019-11-11 21:16:12 +0000 UTC - event for csi-hostpathplugin: {statefulset-controller } SuccessfulDelete: delete Pod csi-hostpathplugin-0 in StatefulSet csi-hostpathplugin successful Nov 11 21:20:54.000: INFO: At 2019-11-11 21:16:12 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:20:54.000: INFO: At 2019-11-11 21:16:22 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:20:54.000: INFO: At 2019-11-11 21:16:33 +0000 UTC - event for csi-hostpath-attacher-0: {kubelet k8s-agentpool-35271922-vmss000000} Pulling: Pulling image "quay.io/k8scsi/csi-attacher:v1.2.0" Nov 11 21:20:54.000: INFO: At 2019-11-11 21:16:34 +0000 UTC - event for csi-hostpath-provisioner-0: {kubelet k8s-agentpool-35271922-vmss000000} Pulling: Pulling image "quay.io/k8scsi/csi-provisioner:v1.4.0-rc1" Nov 11 21:20:54.000: INFO: At 2019-11-11 21:16:34 +0000 UTC - event for csi-hostpath-resizer-0: {kubelet k8s-agentpool-35271922-vmss000000} Pulling: Pulling image "quay.io/k8scsi/csi-resizer:v0.2.0" Nov 11 21:20:54.000: INFO: At 2019-11-11 21:16:34 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:20:54.000: INFO: At 2019-11-11 21:16:34 +0000 UTC - event for csi-snapshotter-0: {kubelet k8s-agentpool-35271922-vmss000000} Pulling: Pulling image "quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1" Nov 11 21:20:54.000: INFO: At 2019-11-11 21:16:35 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:20:54.000: INFO: At 2019-11-11 21:16:46 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:20:54.000: INFO: At 2019-11-11 21:16:52 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:20:54.000: INFO: At 2019-11-11 21:16:56 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:20:54.000: INFO: At 2019-11-11 21:16:57 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:20:54.000: INFO: At 2019-11-11 21:17:06 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:20:54.000: INFO: At 2019-11-11 21:17:09 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:20:54.000: INFO: At 2019-11-11 21:17:09 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:20:54.000: INFO: At 2019-11-11 21:17:13 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:20:54.000: INFO: At 2019-11-11 21:17:17 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:20:54.000: INFO: At 2019-11-11 21:17:18 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:20:54.000: INFO: At 2019-11-11 21:17:23 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:20:54.000: INFO: At 2019-11-11 21:17:24 +0000 UTC - event for csi-hostpath-attacher-0: {kubelet k8s-agentpool-35271922-vmss000000} Pulled: Successfully pulled image "quay.io/k8scsi/csi-attacher:v1.2.0" Nov 11 21:20:54.000: INFO: At 2019-11-11 21:17:27 +0000 UTC - event for csi-hostpath-provisioner-0: {kubelet k8s-agentpool-35271922-vmss000000} Pulled: Successfully pulled image "quay.io/k8scsi/csi-provisioner:v1.4.0-rc1" Nov 11 21:20:54.000: INFO: At 2019-11-11 21:17:28 +0000 UTC - event for csi-hostpath-attacher-0: {kubelet k8s-agentpool-35271922-vmss000000} Created: Created container csi-attacher Nov 11 21:20:54.000: INFO: At 2019-11-11 21:17:28 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:20:54.000: INFO: At 2019-11-11 21:17:30 +0000 UTC - event for csi-snapshotter-0: {kubelet k8s-agentpool-35271922-vmss000000} Pulled: Successfully pulled image "quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1" Nov 11 21:20:54.000: INFO: At 2019-11-11 21:17:31 +0000 UTC - event for csi-hostpath-provisioner-0: {kubelet k8s-agentpool-35271922-vmss000000} Created: Created container csi-provisioner Nov 11 21:20:54.000: INFO: At 2019-11-11 21:17:33 +0000 UTC - event for csi-hostpath-attacher-0: {kubelet k8s-agentpool-35271922-vmss000000} Started: Started container csi-attacher Nov 11 21:20:54.000: INFO: At 2019-11-11 21:17:34 +0000 UTC - event for csi-hostpath-provisioner-0: {kubelet k8s-agentpool-35271922-vmss000000} Started: Started container csi-provisioner Nov 11 21:20:54.000: INFO: At 2019-11-11 21:17:34 +0000 UTC - event for csi-hostpath-resizer-0: {kubelet k8s-agentpool-35271922-vmss000000} Pulled: Successfully pulled image "quay.io/k8scsi/csi-resizer:v0.2.0" Nov 11 21:20:54.000: INFO: At 2019-11-11 21:17:34 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:20:54.000: INFO: At 2019-11-11 21:17:34 +0000 UTC - event for csi-snapshotter-0: {kubelet k8s-agentpool-35271922-vmss000000} Created: Created container csi-snapshotter Nov 11 21:20:54.000: INFO: At 2019-11-11 21:17:38 +0000 UTC - event for csi-hostpath-resizer-0: {kubelet k8s-agentpool-35271922-vmss000000} Created: Created container csi-resizer Nov 11 21:20:54.000: INFO: At 2019-11-11 21:17:38 +0000 UTC - event for csi-snapshotter-0: {kubelet k8s-agentpool-35271922-vmss000000} Started: Started container csi-snapshotter Nov 11 21:20:54.000: INFO: At 2019-11-11 21:17:41 +0000 UTC - event for csi-hostpath-resizer-0: {kubelet k8s-agentpool-35271922-vmss000000} Started: Started container csi-resizer Nov 11 21:20:54.000: INFO: At 2019-11-11 21:17:41 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:20:54.000: INFO: At 2019-11-11 21:17:49 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:20:54.000: INFO: At 2019-11-11 21:17:55 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:20:54.000: INFO: At 2019-11-11 21:18:01 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:20:54.000: INFO: At 2019-11-11 21:18:08 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:20:54.000: INFO: At 2019-11-11 21:18:13 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:20:54.000: INFO: At 2019-11-11 21:18:26 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:20:54.000: INFO: At 2019-11-11 21:18:36 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:20:54.000: INFO: At 2019-11-11 21:18:47 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:20:54.000: INFO: At 2019-11-11 21:18:55 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:20:54.000: INFO: At 2019-11-11 21:18:58 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:20:54.000: INFO: At 2019-11-11 21:19:00 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:20:54.000: INFO: At 2019-11-11 21:19:06 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:20:54.000: INFO: At 2019-11-11 21:19:11 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:20:54.000: INFO: At 2019-11-11 21:19:18 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:20:54.000: INFO: At 2019-11-11 21:19:27 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:20:54.000: INFO: At 2019-11-11 21:19:48 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:20:54.000: INFO: At 2019-11-11 21:20:11 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} Pulling: Pulling image "quay.io/k8scsi/csi-node-driver-registrar:v1.1.0" Nov 11 21:20:54.000: INFO: At 2019-11-11 21:20:26 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} Pulled: Successfully pulled image "quay.io/k8scsi/csi-node-driver-registrar:v1.1.0" Nov 11 21:20:54.000: INFO: At 2019-11-11 21:20:33 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} Created: Created container node-driver-registrar Nov 11 21:20:54.000: INFO: At 2019-11-11 21:20:39 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} Pulling: Pulling image "quay.io/k8scsi/hostpathplugin:v1.2.0-rc5" Nov 11 21:20:54.000: INFO: At 2019-11-11 21:20:39 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} Started: Started container node-driver-registrar Nov 11 21:20:54.000: INFO: At 2019-11-11 21:20:43 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} Pulled: Successfully pulled image "quay.io/k8scsi/hostpathplugin:v1.2.0-rc5" Nov 11 21:20:54.000: INFO: At 2019-11-11 21:20:52 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} Created: Created container hostpath Nov 11 21:20:54.053: INFO: POD NODE PHASE GRACE CONDITIONS Nov 11 21:20:54.053: INFO: csi-hostpath-attacher-0 k8s-agentpool-35271922-vmss000000 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:15:59 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:17:38 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:17:38 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:15:59 +0000 UTC }] Nov 11 21:20:54.053: INFO: csi-hostpath-provisioner-0 k8s-agentpool-35271922-vmss000000 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:16:13 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:17:38 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:17:38 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:16:13 +0000 UTC }] Nov 11 21:20:54.053: INFO: csi-hostpath-resizer-0 k8s-agentpool-35271922-vmss000000 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:16:14 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:17:44 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:17:44 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:16:14 +0000 UTC }] Nov 11 21:20:54.053: INFO: csi-hostpathplugin-0 k8s-agentpool-35271922-vmss000000 Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:19:59 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:19:59 +0000 UTC ContainersNotReady containers with unready status: [node-driver-registrar hostpath liveness-probe]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:19:59 +0000 UTC ContainersNotReady containers with unready status: [node-driver-registrar hostpath liveness-probe]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:19:59 +0000 UTC }] Nov 11 21:20:54.053: INFO: csi-snapshotter-0 k8s-agentpool-35271922-vmss000000 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:16:15 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:17:41 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:17:41 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:16:15 +0000 UTC }] Nov 11 21:20:54.053: INFO: Nov 11 21:20:54.211: INFO: Logging node info for node k8s-agentpool-35271922-vmss000000 Nov 11 21:20:54.263: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool-35271922-vmss000000 /api/v1/nodes/k8s-agentpool-35271922-vmss000000 04da751e-f23a-470e-b29d-e591113d7271 50188 0 2019-11-11 20:48:07 +0000 UTC <nil> <nil> map[agentpool:agentpool beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool-35271922-vmss000000 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: storageprofile:managed storagetier:Premium_LRS] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-2030":"k8s-agentpool-35271922-vmss000000","csi-hostpath-ephemeral-7436":"k8s-agentpool-35271922-vmss000000","csi-hostpath-provisioning-3597":"k8s-agentpool-35271922-vmss000000","csi-hostpath-provisioning-9334":"k8s-agentpool-35271922-vmss000000","csi-hostpath-volume-5092":"k8s-agentpool-35271922-vmss000000","csi-hostpath-volume-expand-1449":"k8s-agentpool-35271922-vmss000000","csi-hostpath-volume-expand-5724":"k8s-agentpool-35271922-vmss000000","csi-mock-csi-mock-volumes-1829":"csi-mock-csi-mock-volumes-1829","csi-mock-csi-mock-volumes-2481":"csi-mock-csi-mock-volumes-2481","csi-mock-csi-mock-volumes-3755":"csi-mock-csi-mock-volumes-3755","csi-mock-csi-mock-volumes-763":"csi-mock-csi-mock-volumes-763","csi-mock-csi-mock-volumes-9721":"csi-mock-csi-mock-volumes-9721"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-35271922-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16797044736 0} {<nil>} 16403364Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16010612736 0} {<nil>} 15635364Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:20:04 +0000 UTC,LastTransitionTime:2019-11-11 20:47:58 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:20:04 +0000 UTC,LastTransitionTime:2019-11-11 20:47:58 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:20:04 +0000 UTC,LastTransitionTime:2019-11-11 20:47:58 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:20:04 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.248.0.4,},NodeAddress{Type:Hostname,Address:k8s-agentpool-35271922-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:84c06f3ee41a47aaa558c81f34711895,SystemUUID:9BB24974-637B-EE4C-A221-0F74BB743424,BootID:59f7eb0c-c09c-4eae-a14b-0710f9e14004,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[quay.io/kubernetes_incubator/nfs-provisioner@sha256:df762117e3c891f2d2ddff46ecb0776ba1f9f3c44cfd7739b0683bcd7a7954a8 quay.io/kubernetes_incubator/nfs-provisioner:v2.2.2],SizeBytes:391772778,},ContainerImage{Names:[gluster/glusterdynamic-provisioner@sha256:90067cb05a7d217651e84576935934fd9dcff8e6bcdcbaa416bbf36fcd09dbd1 gluster/glusterdynamic-provisioner:v1.0],SizeBytes:373281573,},ContainerImage{Names:[gcr.io/google-samples/gb-frontend@sha256:35cb427341429fac3df10ff74600ea73e8ec0754d78f9ce89e0b4f3d70d53ba6 gcr.io/google-samples/gb-frontend:v6],SizeBytes:373099368,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/volume/nfs@sha256:c2ad734346f608a5f7d69cfded93c4e8094069320657bd372d12ba21dea3ea71 gcr.io/kubernetes-e2e-test-images/volume/nfs:1.0],SizeBytes:225358913,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/jessie-dnsutils@sha256:ad583e33cb284f7ef046673809b146ec4053cda19b54a85d2b180a86169715eb gcr.io/kubernetes-e2e-test-images/jessie-dnsutils:1.0],SizeBytes:195659796,},ContainerImage{Names:[httpd@sha256:addd70e4ee83f3bc9a4c1c7c41e37927ba47faf639312fc936df3afad7926f5a httpd:2.4.39-alpine],SizeBytes:126894770,},ContainerImage{Names:[httpd@sha256:eb8ccf084cf3e80eece1add239effefd171eb39adbc154d33c14260d905d4060 httpd:2.4.38-alpine],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1],SizeBytes:121711221,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727 gcr.io/kubernetes-e2e-test-images/agnhost:2.6],SizeBytes:57345321,},ContainerImage{Names:[quay.io/k8scsi/csi-provisioner@sha256:0efcb424f1dde9b9fb11a1a14f2e48ab47e1c3f08bc3a929990dcfcb1f7ab34f quay.io/k8scsi/csi-provisioner:v1.4.0-rc1],SizeBytes:54431016,},ContainerImage{Names:[quay.io/k8scsi/csi-snapshotter@sha256:e3d3e742e32d00488fdb401045b9b1d033d7ca0ab6e760f77b24750fc95e5f70 quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1],SizeBytes:51703561,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0],SizeBytes:46226754,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:0aba670b4d9d6b2e720bbf575d733156c676b693ca26501235444490300db838 quay.io/k8scsi/csi-attacher:v1.1.0],SizeBytes:42839085,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:7d46fb6eb8b890dc546029d1565d502b4a1d974d33625c6ee2bc7991b77fc1a1 quay.io/k8scsi/csi-resizer:v0.2.0],SizeBytes:42817100,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:f315c9042e56def3c05c6b04fe79ec9da6d39ddc557ca365a76cf35964ea08b6 quay.io/k8scsi/csi-resizer:v0.1.0],SizeBytes:42623056,},ContainerImage{Names:[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0],SizeBytes:42488424,},ContainerImage{Names:[redis@sha256:50899ea1ceed33fa03232f3ac57578a424faa1742c1ac9c7a7bdb95cdf19b858 redis:5.0.5-alpine],SizeBytes:29331594,},ContainerImage{Names:[quay.io/k8scsi/hostpathplugin@sha256:b4826e492fc1762fceaf9726f41575ca0a4567864d3d235da874818de18039de quay.io/k8scsi/hostpathplugin:v1.2.0-rc5],SizeBytes:28761497,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7],SizeBytes:22909487,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/echoserver@sha256:e9ba514b896cdf559eef8788b66c2c3ee55f3572df617647b4b0d8b6bf81cf19 gcr.io/kubernetes-e2e-test-images/echoserver:2.2],SizeBytes:21692741,},ContainerImage{Names:[quay.io/k8scsi/mock-driver@sha256:e0eed916b7d970bad2b7d9875f9ad16932f987f0f3d91ec5d86da68b0b5cc9d1 quay.io/k8scsi/mock-driver:v2.1.0],SizeBytes:16226335,},ContainerImage{Names:[nginx@sha256:485b610fefec7ff6c463ced9623314a04ed67e3945b9c08d7e53a47f6d108dc7 nginx:1.14-alpine],SizeBytes:16032814,},ContainerImage{Names:[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0],SizeBytes:15815995,},ContainerImage{Names:[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0],SizeBytes:14967303,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/dnsutils@sha256:2abeee84efb79c14d731966e034af33bf324d3b26ca28497555511ff094b3ddd gcr.io/kubernetes-e2e-test-images/dnsutils:1.1],SizeBytes:9349974,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nautilus@sha256:33a732d4c42a266912a5091598a0f07653c9134db4b8d571690d8afd509e0bfc gcr.io/kubernetes-e2e-test-images/nautilus:1.0],SizeBytes:4753501,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/kitten@sha256:bcbc4875c982ab39aa7c4f6acf4a287f604e996d9f34a3fbda8c3d1a7457d1f6 gcr.io/kubernetes-e2e-test-images/kitten:1.0],SizeBytes:4747037,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/test-webserver@sha256:7f93d6e32798ff28bc6289254d0c2867fe2c849c8e46edc50f8624734309812e gcr.io/kubernetes-e2e-test-images/test-webserver:1.0],SizeBytes:4732240,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 gcr.io/kubernetes-e2e-test-images/mounttest:1.0],SizeBytes:1563521,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29],SizeBytes:1154361,},ContainerImage{Names:[busybox@sha256:bbc3a03235220b170ba48a157dd097dd1379299370e1ed99ce976df0355d24f0 busybox:1.27],SizeBytes:1129289,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea k8s.gcr.io/pause-amd64:3.1 k8s.gcr.io/pause:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:20:54.264: INFO: Logging kubelet events for node k8s-agentpool-35271922-vmss000000 Nov 11 21:20:54.319: INFO: Logging pods the kubelet thinks is on node k8s-agentpool-35271922-vmss000000 Nov 11 21:20:54.448: INFO: csi-snapshotter-0 started at 2019-11-11 21:16:15 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:54.449: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 11 21:20:54.449: INFO: csi-hostpath-attacher-0 started at 2019-11-11 21:19:39 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:54.449: INFO: Container csi-attacher ready: true, restart count 0 Nov 11 21:20:54.449: INFO: csi-hostpath-provisioner-0 started at 2019-11-11 21:14:50 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:54.449: INFO: Container csi-provisioner ready: true, restart count 0 Nov 11 21:20:54.449: INFO: csi-hostpath-attacher-0 started at 2019-11-11 21:18:12 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:54.449: INFO: Container csi-attacher ready: true, restart count 0 Nov 11 21:20:54.449: INFO: pvc-datasource-writer-npqfs started at 2019-11-11 21:19:45 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:54.449: INFO: Container volume-tester ready: false, restart count 0 Nov 11 21:20:54.449: INFO: csi-hostpathplugin-0 started at 2019-11-11 21:20:52 +0000 UTC (0+0 container statuses recorded) Nov 11 21:20:54.449: INFO: kubernetes-dashboard-65966766b9-lv2x8 started at 2019-11-11 20:48:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:54.449: INFO: Container kubernetes-dashboard ready: true, restart count 0 Nov 11 21:20:54.449: INFO: local-injector started at 2019-11-11 21:20:45 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:54.449: INFO: Container local-injector ready: false, restart count 0 Nov 11 21:20:54.449: INFO: update-demo-kitten-tnn79 started at 2019-11-11 21:19:06 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:54.449: INFO: Container update-demo ready: true, restart count 0 Nov 11 21:20:54.449: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:19:28 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:54.449: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:20:54.449: INFO: security-context-26615253-e5ed-4815-a08f-9561bc160325 started at 2019-11-11 21:20:42 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:54.449: INFO: Container write-pod ready: false, restart count 0 Nov 11 21:20:54.449: INFO: csi-snapshotter-0 started at 2019-11-11 21:14:52 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:54.449: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 11 21:20:54.449: INFO: csi-hostpath-provisioner-0 started at 2019-11-11 21:19:42 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:54.449: INFO: Container csi-provisioner ready: true, restart count 0 Nov 11 21:20:54.449: INFO: csi-hostpath-provisioner-0 started at 2019-11-11 21:15:11 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:54.449: INFO: Container csi-provisioner ready: false, restart count 0 Nov 11 21:20:54.449: INFO: hostpath-symlink-prep-volume-593 started at 2019-11-11 21:19:45 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:54.449: INFO: Container init-volume-volume-593 ready: false, restart count 0 Nov 11 21:20:54.449: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:20:43 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:54.449: INFO: Container agnhost ready: false, restart count 0 Nov 11 21:20:54.449: INFO: csi-hostpath-provisioner-0 started at 2019-11-11 21:16:13 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:54.449: INFO: Container csi-provisioner ready: true, restart count 0 Nov 11 21:20:54.449: INFO: csi-hostpathplugin-0 started at 2019-11-11 21:20:43 +0000 UTC (0+0 container statuses recorded) Nov 11 21:20:54.449: INFO: inline-volume-tester-v8qbq started at 2019-11-11 21:14:53 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:54.449: INFO: Container csi-volume-tester ready: false, restart count 0 Nov 11 21:20:54.449: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:19:35 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:54.449: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:20:54.449: INFO: csi-hostpath-resizer-0 started at 2019-11-11 21:19:42 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:54.449: INFO: Container csi-resizer ready: true, restart count 0 Nov 11 21:20:54.449: INFO: csi-hostpath-resizer-0 started at 2019-11-11 21:18:13 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:54.449: INFO: Container csi-resizer ready: true, restart count 0 Nov 11 21:20:54.449: INFO: csi-snapshotter-0 started at 2019-11-11 21:18:13 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:54.449: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 11 21:20:54.449: INFO: blobfuse-flexvol-installer-dmnhw started at 2019-11-11 20:48:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:54.449: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 Nov 11 21:20:54.449: INFO: csi-hostpath-provisioner-0 started at 2019-11-11 21:18:13 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:54.449: INFO: Container csi-provisioner ready: true, restart count 0 Nov 11 21:20:54.449: INFO: csi-hostpath-attacher-0 started at 2019-11-11 21:15:08 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:54.449: INFO: Container csi-attacher ready: false, restart count 0 Nov 11 21:20:54.449: INFO: csi-hostpath-resizer-0 started at 2019-11-11 21:16:14 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:54.449: INFO: Container csi-resizer ready: true, restart count 0 Nov 11 21:20:54.449: INFO: coredns-87f5d796-6mz2l started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:54.449: INFO: Container coredns ready: true, restart count 0 Nov 11 21:20:54.449: INFO: csi-hostpath-resizer-0 started at 2019-11-11 21:15:12 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:54.449: INFO: Container csi-resizer ready: false, restart count 0 Nov 11 21:20:54.449: INFO: csi-hostpath-attacher-0 started at 2019-11-11 21:19:23 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:54.449: INFO: Container csi-attacher ready: true, restart count 0 Nov 11 21:20:54.449: INFO: update-demo-nautilus-fsljl started at 2019-11-11 21:17:03 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:54.449: INFO: Container update-demo ready: false, restart count 0 Nov 11 21:20:54.449: INFO: csi-hostpath-attacher-0 started at 2019-11-11 21:15:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:54.449: INFO: Container csi-attacher ready: true, restart count 0 Nov 11 21:20:54.449: INFO: csi-snapshotter-0 started at 2019-11-11 21:19:44 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:54.449: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 11 21:20:54.449: INFO: kube-proxy-nwdpz started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:54.449: INFO: Container kube-proxy ready: true, restart count 0 Nov 11 21:20:54.449: INFO: csi-hostpath-attacher-0 started at 2019-11-11 21:14:46 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:54.449: INFO: Container csi-attacher ready: true, restart count 0 Nov 11 21:20:54.449: INFO: csi-hostpath-provisioner-0 started at 2019-11-11 21:19:23 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:54.449: INFO: Container csi-provisioner ready: true, restart count 0 Nov 11 21:20:54.449: INFO: csi-hostpath-resizer-0 started at 2019-11-11 21:19:24 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:54.449: INFO: Container csi-resizer ready: true, restart count 0 Nov 11 21:20:54.449: INFO: externalsvc-fkjkw started at 2019-11-11 21:18:13 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:54.449: INFO: Container externalsvc ready: false, restart count 0 Nov 11 21:20:54.449: INFO: csi-hostpathplugin-0 started at 2019-11-11 21:19:59 +0000 UTC (0+3 container statuses recorded) Nov 11 21:20:54.449: INFO: Container hostpath ready: false, restart count 0 Nov 11 21:20:54.449: INFO: Container liveness-probe ready: false, restart count 0 Nov 11 21:20:54.449: INFO: Container node-driver-registrar ready: false, restart count 0 Nov 11 21:20:54.449: INFO: csi-hostpathplugin-0 started at 2019-11-11 21:20:41 +0000 UTC (0+0 container statuses recorded) Nov 11 21:20:54.449: INFO: csi-hostpathplugin-0 started at 2019-11-11 21:20:37 +0000 UTC (0+0 container statuses recorded) Nov 11 21:20:54.449: INFO: azure-ip-masq-agent-c4mb2 started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:54.449: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:20:54.449: INFO: csi-snapshotter-0 started at 2019-11-11 21:15:13 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:54.449: INFO: Container csi-snapshotter ready: false, restart count 0 Nov 11 21:20:54.449: INFO: csi-mockplugin-0 started at 2019-11-11 21:19:31 +0000 UTC (0+3 container statuses recorded) Nov 11 21:20:54.449: INFO: Container csi-provisioner ready: false, restart count 0 Nov 11 21:20:54.449: INFO: Container driver-registrar ready: false, restart count 0 Nov 11 21:20:54.449: INFO: Container mock ready: false, restart count 0 Nov 11 21:20:54.449: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:20:13 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:54.449: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:20:54.449: INFO: csi-hostpath-resizer-0 started at 2019-11-11 21:14:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:54.449: INFO: Container csi-resizer ready: true, restart count 0 Nov 11 21:20:54.449: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:20:38 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:54.449: INFO: Container agnhost ready: false, restart count 0 Nov 11 21:20:54.449: INFO: pod-subpath-test-local-preprovisionedpv-s2mt started at 2019-11-11 21:20:47 +0000 UTC (2+1 container statuses recorded) Nov 11 21:20:54.449: INFO: Init container init-volume-local-preprovisionedpv-s2mt ready: false, restart count 0 Nov 11 21:20:54.449: INFO: Init container test-init-volume-local-preprovisionedpv-s2mt ready: false, restart count 0 Nov 11 21:20:54.449: INFO: Container test-container-subpath-local-preprovisionedpv-s2mt ready: false, restart count 0 Nov 11 21:20:54.449: INFO: keyvault-flexvolume-zzp7p started at 2019-11-11 20:48:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:54.449: INFO: Container keyvault-flexvolume ready: true, restart count 0 Nov 11 21:20:54.449: INFO: csi-snapshotter-0 started at 2019-11-11 21:19:24 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:54.449: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 11 21:20:54.449: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:19:49 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:54.449: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:20:54.449: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:20:33 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:54.449: INFO: Container agnhost ready: false, restart count 0 W1111 21:20:54.503038 93386 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:20:55.666: INFO: Latency metrics for node k8s-agentpool-35271922-vmss000000 Nov 11 21:20:55.666: INFO: Logging node info for node k8s-agentpool-35271922-vmss000001 Nov 11 21:20:55.719: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool-35271922-vmss000001 /api/v1/nodes/k8s-agentpool-35271922-vmss000001 af5ba0b6-4f7f-4468-9d3c-b3e116ca81ca 50862 0 2019-11-11 20:48:13 +0000 UTC <nil> <nil> map[agentpool:agentpool beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool-35271922-vmss000001 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: storageprofile:managed storagetier:Premium_LRS] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-8406":"k8s-agentpool-35271922-vmss000001","csi-hostpath-provisioning-2466":"k8s-agentpool-35271922-vmss000001","csi-hostpath-provisioning-265":"k8s-agentpool-35271922-vmss000001","csi-hostpath-provisioning-310":"k8s-agentpool-35271922-vmss000001","csi-hostpath-provisioning-753":"k8s-agentpool-35271922-vmss000001","csi-hostpath-volume-expand-3983":"k8s-agentpool-35271922-vmss000001","csi-mock-csi-mock-volumes-2781":"csi-mock-csi-mock-volumes-2781","csi-mock-csi-mock-volumes-2838":"csi-mock-csi-mock-volumes-2838","csi-mock-csi-mock-volumes-3306":"csi-mock-csi-mock-volumes-3306","csi-mock-csi-mock-volumes-6769":"csi-mock-csi-mock-volumes-6769","csi-mock-csi-mock-volumes-694":"csi-mock-csi-mock-volumes-694","csi-mock-csi-mock-volumes-797":"csi-mock-csi-mock-volumes-797"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-35271922-vmss/virtualMachines/1,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16797044736 0} {<nil>} 16403364Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16010612736 0} {<nil>} 15635364Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:20:29 +0000 UTC,LastTransitionTime:2019-11-11 20:48:13 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:20:29 +0000 UTC,LastTransitionTime:2019-11-11 20:48:13 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:20:29 +0000 UTC,LastTransitionTime:2019-11-11 20:48:13 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:20:29 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.248.0.5,},NodeAddress{Type:Hostname,Address:k8s-agentpool-35271922-vmss000001,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:1df8c22200b84e64b76150d232ecb929,SystemUUID:B03EE801-7A2D-7140-BE20-B2C844575A73,BootID:d32206f7-48b0-46c5-a542-39721566b456,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[quay.io/kubernetes_incubator/nfs-provisioner@sha256:df762117e3c891f2d2ddff46ecb0776ba1f9f3c44cfd7739b0683bcd7a7954a8 quay.io/kubernetes_incubator/nfs-provisioner:v2.2.2],SizeBytes:391772778,},ContainerImage{Names:[gcr.io/google-samples/gb-frontend@sha256:35cb427341429fac3df10ff74600ea73e8ec0754d78f9ce89e0b4f3d70d53ba6 gcr.io/google-samples/gb-frontend:v6],SizeBytes:373099368,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/volume/nfs@sha256:c2ad734346f608a5f7d69cfded93c4e8094069320657bd372d12ba21dea3ea71 gcr.io/kubernetes-e2e-test-images/volume/nfs:1.0],SizeBytes:225358913,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/jessie-dnsutils@sha256:ad583e33cb284f7ef046673809b146ec4053cda19b54a85d2b180a86169715eb gcr.io/kubernetes-e2e-test-images/jessie-dnsutils:1.0],SizeBytes:195659796,},ContainerImage{Names:[httpd@sha256:addd70e4ee83f3bc9a4c1c7c41e37927ba47faf639312fc936df3afad7926f5a httpd:2.4.39-alpine],SizeBytes:126894770,},ContainerImage{Names:[httpd@sha256:eb8ccf084cf3e80eece1add239effefd171eb39adbc154d33c14260d905d4060 httpd:2.4.38-alpine],SizeBytes:123781643,},ContainerImage{Names:[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0],SizeBytes:71821984,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727 gcr.io/kubernetes-e2e-test-images/agnhost:2.6],SizeBytes:57345321,},ContainerImage{Names:[quay.io/k8scsi/csi-provisioner@sha256:0efcb424f1dde9b9fb11a1a14f2e48ab47e1c3f08bc3a929990dcfcb1f7ab34f quay.io/k8scsi/csi-provisioner:v1.4.0-rc1],SizeBytes:54431016,},ContainerImage{Names:[quay.io/k8scsi/csi-snapshotter@sha256:e3d3e742e32d00488fdb401045b9b1d033d7ca0ab6e760f77b24750fc95e5f70 quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1],SizeBytes:51703561,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0],SizeBytes:46226754,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:0aba670b4d9d6b2e720bbf575d733156c676b693ca26501235444490300db838 quay.io/k8scsi/csi-attacher:v1.1.0],SizeBytes:42839085,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:7d46fb6eb8b890dc546029d1565d502b4a1d974d33625c6ee2bc7991b77fc1a1 quay.io/k8scsi/csi-resizer:v0.2.0],SizeBytes:42817100,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:f315c9042e56def3c05c6b04fe79ec9da6d39ddc557ca365a76cf35964ea08b6 quay.io/k8scsi/csi-resizer:v0.1.0],SizeBytes:42623056,},ContainerImage{Names:[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1],SizeBytes:42541759,},ContainerImage{Names:[gcr.io/google-containers/debian-base@sha256:6966a0aedd7592c18ff2dd803c08bd85780ee19f5e3a2e7cf908a4cd837afcde gcr.io/google-containers/debian-base:0.4.1],SizeBytes:42323657,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nonroot@sha256:d4ede5c74517090b6686219059118ed178cf4620f5db8781b32f806bb1e7395b gcr.io/kubernetes-e2e-test-images/nonroot:1.0],SizeBytes:42321438,},ContainerImage{Names:[redis@sha256:50899ea1ceed33fa03232f3ac57578a424faa1742c1ac9c7a7bdb95cdf19b858 redis:5.0.5-alpine],SizeBytes:29331594,},ContainerImage{Names:[quay.io/k8scsi/hostpathplugin@sha256:b4826e492fc1762fceaf9726f41575ca0a4567864d3d235da874818de18039de quay.io/k8scsi/hostpathplugin:v1.2.0-rc5],SizeBytes:28761497,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7],SizeBytes:22909487,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/echoserver@sha256:e9ba514b896cdf559eef8788b66c2c3ee55f3572df617647b4b0d8b6bf81cf19 gcr.io/kubernetes-e2e-test-images/echoserver:2.2],SizeBytes:21692741,},ContainerImage{Names:[quay.io/k8scsi/mock-driver@sha256:e0eed916b7d970bad2b7d9875f9ad16932f987f0f3d91ec5d86da68b0b5cc9d1 quay.io/k8scsi/mock-driver:v2.1.0],SizeBytes:16226335,},ContainerImage{Names:[nginx@sha256:485b610fefec7ff6c463ced9623314a04ed67e3945b9c08d7e53a47f6d108dc7 nginx:1.14-alpine],SizeBytes:16032814,},ContainerImage{Names:[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0],SizeBytes:15815995,},ContainerImage{Names:[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0],SizeBytes:14967303,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/dnsutils@sha256:2abeee84efb79c14d731966e034af33bf324d3b26ca28497555511ff094b3ddd gcr.io/kubernetes-e2e-test-images/dnsutils:1.1],SizeBytes:9349974,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nonewprivs@sha256:10066e9039219449fe3c81f38fe01928f87914150768ab81b62a468e51fa7411 gcr.io/kubernetes-e2e-test-images/nonewprivs:1.0],SizeBytes:6757579,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nautilus@sha256:33a732d4c42a266912a5091598a0f07653c9134db4b8d571690d8afd509e0bfc gcr.io/kubernetes-e2e-test-images/nautilus:1.0],SizeBytes:4753501,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/kitten@sha256:bcbc4875c982ab39aa7c4f6acf4a287f604e996d9f34a3fbda8c3d1a7457d1f6 gcr.io/kubernetes-e2e-test-images/kitten:1.0],SizeBytes:4747037,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/test-webserver@sha256:7f93d6e32798ff28bc6289254d0c2867fe2c849c8e46edc50f8624734309812e gcr.io/kubernetes-e2e-test-images/test-webserver:1.0],SizeBytes:4732240,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:4206620,},ContainerImage{Names:[alpine@sha256:8421d9a84432575381bfabd248f1eb56f3aa21d9d7cd2511583c68c9b7511d10 alpine:3.7],SizeBytes:4206494,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 gcr.io/kubernetes-e2e-test-images/mounttest:1.0],SizeBytes:1563521,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest-user@sha256:17319ca525ee003681fccf7e8c6b1b910ff4f49b653d939ac7f9b6e7c463933d gcr.io/kubernetes-e2e-test-images/mounttest-user:1.0],SizeBytes:1450451,},ContainerImage{Names:[busybox@sha256:1303dbf110c57f3edf68d9f5a16c082ec06c4cf7604831669faf2c712260b5a0 busybox:latest],SizeBytes:1219790,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29],SizeBytes:1154361,},ContainerImage{Names:[busybox@sha256:bbc3a03235220b170ba48a157dd097dd1379299370e1ed99ce976df0355d24f0 busybox:1.27],SizeBytes:1129289,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea k8s.gcr.io/pause-amd64:3.1 k8s.gcr.io/pause:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:20:55.719: INFO: Logging kubelet events for node k8s-agentpool-35271922-vmss000001 Nov 11 21:20:55.783: INFO: Logging pods the kubelet thinks is on node k8s-agentpool-35271922-vmss000001 Nov 11 21:20:55.890: INFO: csi-hostpath-attacher-0 started at 2019-11-11 21:15:21 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:55.890: INFO: Container csi-attacher ready: true, restart count 0 Nov 11 21:20:55.890: INFO: emptydir-injector started at 2019-11-11 21:20:15 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:55.890: INFO: Container emptydir-injector ready: false, restart count 0 Nov 11 21:20:55.890: INFO: execpod7bqwb started at 2019-11-11 21:19:08 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:55.890: INFO: Container agnhost-pause ready: true, restart count 0 Nov 11 21:20:55.891: INFO: azure-ip-masq-agent-9427k started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:55.891: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:20:55.891: INFO: redis-master-xrh5v started at 2019-11-11 21:20:38 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:55.891: INFO: Container redis-master ready: false, restart count 0 Nov 11 21:20:55.891: INFO: pod-exec-websocket-78564d09-9f05-443d-a805-a8f276b24d9c started at 2019-11-11 21:19:28 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:55.891: INFO: Container main ready: true, restart count 0 Nov 11 21:20:55.891: INFO: pod-adoption-release started at 2019-11-11 21:18:50 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:55.891: INFO: Container pod-adoption-release ready: false, restart count 0 Nov 11 21:20:55.891: INFO: csi-hostpath-provisioner-0 started at 2019-11-11 21:15:26 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:55.891: INFO: Container csi-provisioner ready: true, restart count 0 Nov 11 21:20:55.891: INFO: pvc-tester-cbcvr started at 2019-11-11 21:20:55 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:55.891: INFO: Container write-pod ready: false, restart count 0 Nov 11 21:20:55.891: INFO: blobfuse-flexvol-installer-tvkwl started at 2019-11-11 20:48:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:55.891: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 Nov 11 21:20:55.891: INFO: tiller-deploy-7559b6b885-jsnn2 started at 2019-11-11 20:48:53 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:55.891: INFO: Container tiller ready: true, restart count 0 Nov 11 21:20:55.891: INFO: metrics-server-58ff8c5ddf-thxvp started at 2019-11-11 20:48:53 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:55.891: INFO: Container metrics-server ready: true, restart count 0 Nov 11 21:20:55.891: INFO: update-demo-kitten-9r5pv started at 2019-11-11 21:18:02 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:55.891: INFO: Container update-demo ready: true, restart count 0 Nov 11 21:20:55.891: INFO: csi-hostpathplugin-0 started at 2019-11-11 21:15:25 +0000 UTC (0+3 container statuses recorded) Nov 11 21:20:55.891: INFO: Container hostpath ready: true, restart count 0 Nov 11 21:20:55.891: INFO: Container liveness-probe ready: true, restart count 0 Nov 11 21:20:55.891: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 11 21:20:55.891: INFO: externalsvc-lskpv started at 2019-11-11 21:18:10 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:55.891: INFO: Container externalsvc ready: false, restart count 0 Nov 11 21:20:55.891: INFO: nfs-server started at 2019-11-11 21:18:43 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:55.891: INFO: Container nfs-server ready: true, restart count 0 Nov 11 21:20:55.891: INFO: downwardapi-volume-ef6ebb67-107a-4193-884f-76105d2c23cc started at 2019-11-11 21:20:18 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:55.891: INFO: Container client-container ready: false, restart count 0 Nov 11 21:20:55.891: INFO: csi-hostpath-resizer-0 started at 2019-11-11 21:15:27 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:55.891: INFO: Container csi-resizer ready: true, restart count 0 Nov 11 21:20:55.891: INFO: terminate-cmd-rpa6eaedd83-0bcd-45d5-805e-572e5c947206 started at 2019-11-11 21:20:45 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:55.891: INFO: Container terminate-cmd-rpa ready: false, restart count 0 Nov 11 21:20:55.891: INFO: client-envvars-e30c32e5-7f0b-45fa-839b-2c4681e328bf started at <nil> (0+0 container statuses recorded) Nov 11 21:20:55.891: INFO: keyvault-flexvolume-2s2wq started at 2019-11-11 20:48:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:55.891: INFO: Container keyvault-flexvolume ready: true, restart count 0 Nov 11 21:20:55.891: INFO: downward-api-f2099c57-5d23-4ad5-bfb2-e3287e7289b4 started at 2019-11-11 21:20:16 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:55.891: INFO: Container dapi-container ready: false, restart count 0 Nov 11 21:20:55.891: INFO: csi-snapshotter-0 started at 2019-11-11 21:15:28 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:55.891: INFO: Container csi-snapshotter ready: false, restart count 0 Nov 11 21:20:55.891: INFO: ss-0 started at 2019-11-11 21:19:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:55.891: INFO: Container webserver ready: false, restart count 0 Nov 11 21:20:55.891: INFO: server-envvars-e812e0b1-5cea-439f-89c6-0b9441de526b started at 2019-11-11 21:20:20 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:55.891: INFO: Container srv ready: true, restart count 0 Nov 11 21:20:55.891: INFO: kube-proxy-sgvk2 started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:55.891: INFO: Container kube-proxy ready: true, restart count 0 W1111 21:20:55.945111 93386 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:20:56.493: INFO: Latency metrics for node k8s-agentpool-35271922-vmss000001 Nov 11 21:20:56.494: INFO: Logging node info for node k8s-master-35271922-vmss000000 Nov 11 21:20:56.546: INFO: Node Info: &Node{ObjectMeta:{k8s-master-35271922-vmss000000 /api/v1/nodes/k8s-master-35271922-vmss000000 f042d5b7-5ee8-4528-ae46-c997fef88c1c 50163 0 2019-11-11 20:48:07 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-35271922-vmss000000 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-35271922-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284363264 0} {<nil>} 7113636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6497931264 0} {<nil>} 6345636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:20:02 +0000 UTC,LastTransitionTime:2019-11-11 20:48:04 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:20:02 +0000 UTC,LastTransitionTime:2019-11-11 20:48:04 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:20:02 +0000 UTC,LastTransitionTime:2019-11-11 20:48:04 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:20:02 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.4,},NodeAddress{Type:Hostname,Address:k8s-master-35271922-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:8843d4f5cd0544d08ae501580db6a3a7,SystemUUID:41DCC090-746D-9E44-8888-8A77F0E0C51C,BootID:42d9eb02-ab49-4c2a-8791-7c5b2ac8f2bc,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:45e5b123e6188348715630acd2f7c1469c5b5181078ce90dc6f5399be4c17be1 k8sprow.azurecr.io/azure-cloud-controller-manager:1193984044413489154-9318bf1d],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:20:56.547: INFO: Logging kubelet events for node k8s-master-35271922-vmss000000 Nov 11 21:20:56.603: INFO: Logging pods the kubelet thinks is on node k8s-master-35271922-vmss000000 Nov 11 21:20:56.676: INFO: azure-ip-masq-agent-k7l5b started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:56.676: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:20:56.676: INFO: kube-proxy-8hzbt started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:56.676: INFO: Container kube-proxy ready: true, restart count 0 Nov 11 21:20:56.676: INFO: cloud-controller-manager-k8s-master-35271922-vmss000000 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:56.676: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 11 21:20:56.676: INFO: kube-addon-manager-k8s-master-35271922-vmss000000 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:56.676: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 11 21:20:56.676: INFO: kube-apiserver-k8s-master-35271922-vmss000000 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:56.676: INFO: Container kube-apiserver ready: true, restart count 0 Nov 11 21:20:56.676: INFO: kube-controller-manager-k8s-master-35271922-vmss000000 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:56.676: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 11 21:20:56.676: INFO: kube-scheduler-k8s-master-35271922-vmss000000 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:56.676: INFO: Container kube-scheduler ready: true, restart count 0 W1111 21:20:56.732758 93386 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:20:56.852: INFO: Latency metrics for node k8s-master-35271922-vmss000000 Nov 11 21:20:56.852: INFO: Logging node info for node k8s-master-35271922-vmss000001 Nov 11 21:20:56.903: INFO: Node Info: &Node{ObjectMeta:{k8s-master-35271922-vmss000001 /api/v1/nodes/k8s-master-35271922-vmss000001 0ff06487-2e4c-4c07-a6f2-d77d1a47bbcd 50198 0 2019-11-11 20:48:39 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-35271922-vmss000001 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.6.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-35271922-vmss/virtualMachines/1,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.6.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284363264 0} {<nil>} 7113636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6497931264 0} {<nil>} 6345636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:49:01 +0000 UTC,LastTransitionTime:2019-11-11 20:49:01 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:20:05 +0000 UTC,LastTransitionTime:2019-11-11 20:48:32 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:20:05 +0000 UTC,LastTransitionTime:2019-11-11 20:48:32 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:20:05 +0000 UTC,LastTransitionTime:2019-11-11 20:48:32 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:20:05 +0000 UTC,LastTransitionTime:2019-11-11 20:48:39 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.5,},NodeAddress{Type:Hostname,Address:k8s-master-35271922-vmss000001,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:24b17940193c48e38f59d10d619fb718,SystemUUID:25FDA250-5862-9148-98C8-17FFDD874AEC,BootID:cdd7987a-6ece-464d-a177-21f53ae0b68b,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:45e5b123e6188348715630acd2f7c1469c5b5181078ce90dc6f5399be4c17be1 k8sprow.azurecr.io/azure-cloud-controller-manager:1193984044413489154-9318bf1d],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:20:56.904: INFO: Logging kubelet events for node k8s-master-35271922-vmss000001 Nov 11 21:20:56.961: INFO: Logging pods the kubelet thinks is on node k8s-master-35271922-vmss000001 Nov 11 21:20:57.036: INFO: cloud-controller-manager-k8s-master-35271922-vmss000001 started at 2019-11-11 20:48:26 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:57.036: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 11 21:20:57.036: INFO: kube-addon-manager-k8s-master-35271922-vmss000001 started at 2019-11-11 20:48:26 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:57.036: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 11 21:20:57.036: INFO: kube-apiserver-k8s-master-35271922-vmss000001 started at 2019-11-11 20:48:26 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:57.036: INFO: Container kube-apiserver ready: true, restart count 0 Nov 11 21:20:57.036: INFO: kube-controller-manager-k8s-master-35271922-vmss000001 started at 2019-11-11 20:48:26 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:57.036: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 11 21:20:57.036: INFO: kube-scheduler-k8s-master-35271922-vmss000001 started at 2019-11-11 20:48:26 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:57.036: INFO: Container kube-scheduler ready: true, restart count 0 Nov 11 21:20:57.036: INFO: azure-ip-masq-agent-bm2t5 started at 2019-11-11 20:48:40 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:57.036: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:20:57.036: INFO: kube-proxy-5fcm6 started at 2019-11-11 20:48:40 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:57.036: INFO: Container kube-proxy ready: true, restart count 0 W1111 21:20:57.088915 93386 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:20:57.223: INFO: Latency metrics for node k8s-master-35271922-vmss000001 Nov 11 21:20:57.223: INFO: Logging node info for node k8s-master-35271922-vmss000002 Nov 11 21:20:57.275: INFO: Node Info: &Node{ObjectMeta:{k8s-master-35271922-vmss000002 /api/v1/nodes/k8s-master-35271922-vmss000002 7fedbabf-dce6-48d1-994b-628369e7bcb5 50189 0 2019-11-11 20:48:08 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-35271922-vmss000002 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.4.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-35271922-vmss/virtualMachines/2,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284363264 0} {<nil>} 7113636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6497931264 0} {<nil>} 6345636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:20:04 +0000 UTC,LastTransitionTime:2019-11-11 20:48:03 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:20:04 +0000 UTC,LastTransitionTime:2019-11-11 20:48:03 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:20:04 +0000 UTC,LastTransitionTime:2019-11-11 20:48:03 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:20:04 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.6,},NodeAddress{Type:Hostname,Address:k8s-master-35271922-vmss000002,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:5ecd4c723b8845b18d85d341ba062204,SystemUUID:9E772AEA-D3D7-084D-98ED-1C1A511168B4,BootID:a73a4655-81c3-42e2-9d20-ef7e3cff3664,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:45e5b123e6188348715630acd2f7c1469c5b5181078ce90dc6f5399be4c17be1 k8sprow.azurecr.io/azure-cloud-controller-manager:1193984044413489154-9318bf1d],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:20:57.275: INFO: Logging kubelet events for node k8s-master-35271922-vmss000002 Nov 11 21:20:57.330: INFO: Logging pods the kubelet thinks is on node k8s-master-35271922-vmss000002 Nov 11 21:20:57.410: INFO: kube-proxy-t2sp7 started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:57.410: INFO: Container kube-proxy ready: true, restart count 0 Nov 11 21:20:57.410: INFO: cloud-controller-manager-k8s-master-35271922-vmss000002 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:57.410: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 11 21:20:57.410: INFO: kube-addon-manager-k8s-master-35271922-vmss000002 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:57.410: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 11 21:20:57.410: INFO: kube-apiserver-k8s-master-35271922-vmss000002 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:57.410: INFO: Container kube-apiserver ready: true, restart count 0 Nov 11 21:20:57.410: INFO: kube-controller-manager-k8s-master-35271922-vmss000002 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:57.410: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 11 21:20:57.410: INFO: kube-scheduler-k8s-master-35271922-vmss000002 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:57.410: INFO: Container kube-scheduler ready: true, restart count 0 Nov 11 21:20:57.410: INFO: azure-ip-masq-agent-cmbqm started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:57.410: INFO: Container azure-ip-masq-agent ready: true, restart count 0 W1111 21:20:57.463575 93386 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:20:57.586: INFO: Latency metrics for node k8s-master-35271922-vmss000002 Nov 11 21:20:57.586: INFO: Logging node info for node k8s-master-35271922-vmss000003 Nov 11 21:20:57.639: INFO: Node Info: &Node{ObjectMeta:{k8s-master-35271922-vmss000003 /api/v1/nodes/k8s-master-35271922-vmss000003 d4d04c51-fc7b-4cdc-a1e3-69990a198568 50197 0 2019-11-11 20:48:10 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-35271922-vmss000003 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.5.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-35271922-vmss/virtualMachines/3,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.5.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284363264 0} {<nil>} 7113636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6497931264 0} {<nil>} 6345636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:20:05 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:20:05 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:20:05 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:20:05 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.7,},NodeAddress{Type:Hostname,Address:k8s-master-35271922-vmss000003,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:53117d11ac104f35951d19e4a4448f8f,SystemUUID:048B9870-F8AE-BC47-80D8-26703FED9760,BootID:fc64edd8-d9c5-4d6f-b9e7-c902146bc724,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:45e5b123e6188348715630acd2f7c1469c5b5181078ce90dc6f5399be4c17be1 k8sprow.azurecr.io/azure-cloud-controller-manager:1193984044413489154-9318bf1d],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:20:57.639: INFO: Logging kubelet events for node k8s-master-35271922-vmss000003 Nov 11 21:20:57.695: INFO: Logging pods the kubelet thinks is on node k8s-master-35271922-vmss000003 Nov 11 21:20:57.776: INFO: kube-controller-manager-k8s-master-35271922-vmss000003 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:57.776: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 11 21:20:57.776: INFO: kube-scheduler-k8s-master-35271922-vmss000003 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:57.776: INFO: Container kube-scheduler ready: true, restart count 0 Nov 11 21:20:57.776: INFO: azure-ip-masq-agent-rs668 started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:57.776: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:20:57.776: INFO: kube-proxy-tqjkf started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:57.776: INFO: Container kube-proxy ready: true, restart count 0 Nov 11 21:20:57.776: INFO: cloud-controller-manager-k8s-master-35271922-vmss000003 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:57.776: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 11 21:20:57.776: INFO: kube-addon-manager-k8s-master-35271922-vmss000003 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:57.776: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 11 21:20:57.776: INFO: kube-apiserver-k8s-master-35271922-vmss000003 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:57.776: INFO: Container kube-apiserver ready: true, restart count 0 W1111 21:20:57.828917 93386 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:20:57.952: INFO: Latency metrics for node k8s-master-35271922-vmss000003 Nov 11 21:20:57.952: INFO: Logging node info for node k8s-master-35271922-vmss000004 Nov 11 21:20:58.005: INFO: Node Info: &Node{ObjectMeta:{k8s-master-35271922-vmss000004 /api/v1/nodes/k8s-master-35271922-vmss000004 edf29ad2-2fc8-4411-ac05-e1bf8781e340 50211 0 2019-11-11 20:48:11 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-35271922-vmss000004 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-35271922-vmss/virtualMachines/4,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284363264 0} {<nil>} 7113636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6497931264 0} {<nil>} 6345636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:20:07 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:20:07 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:20:07 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:20:07 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.8,},NodeAddress{Type:Hostname,Address:k8s-master-35271922-vmss000004,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:3ecc95ca67e14e2e87659efc6707e3b4,SystemUUID:0649BB2F-7C03-5743-A309-9BABFF3DD11A,BootID:1256b8d0-bf8e-4490-a45d-beaf41e45a09,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:45e5b123e6188348715630acd2f7c1469c5b5181078ce90dc6f5399be4c17be1 k8sprow.azurecr.io/azure-cloud-controller-manager:1193984044413489154-9318bf1d],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:20:58.005: INFO: Logging kubelet events for node k8s-master-35271922-vmss000004 Nov 11 21:20:58.086: INFO: Logging pods the kubelet thinks is on node k8s-master-35271922-vmss000004 Nov 11 21:20:58.179: INFO: kube-proxy-zrmqq started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:58.179: INFO: Container kube-proxy ready: true, restart count 0 Nov 11 21:20:58.179: INFO: cloud-controller-manager-k8s-master-35271922-vmss000004 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:58.179: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 11 21:20:58.179: INFO: kube-addon-manager-k8s-master-35271922-vmss000004 started at 2019-11-11 20:48:19 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:58.179: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 11 21:20:58.179: INFO: kube-apiserver-k8s-master-35271922-vmss000004 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:58.179: INFO: Container kube-apiserver ready: true, restart count 0 Nov 11 21:20:58.179: INFO: kube-controller-manager-k8s-master-35271922-vmss000004 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:58.179: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 11 21:20:58.179: INFO: kube-scheduler-k8s-master-35271922-vmss000004 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:58.179: INFO: Container kube-scheduler ready: true, restart count 0 Nov 11 21:20:58.179: INFO: azure-ip-masq-agent-trzsm started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:20:58.179: INFO: Container azure-ip-masq-agent ready: true, restart count 0 W1111 21:20:58.232134 93386 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:20:58.362: INFO: Latency metrics for node k8s-master-35271922-vmss000004 Nov 11 21:20:58.362: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "volume-expand-5832" for this suite. Nov 11 21:21:44.579: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered Nov 11 21:21:46.321: INFO: namespace volume-expand-5832 deletion completed in 47.905553125s
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[sig\-storage\]\sCSI\sVolumes\s\[Driver\:\scsi\-hostpath\]\s\[Testpattern\:\sDynamic\sPV\s\(block\svolmode\)\]\svolume\-expand\sshould\snot\sallow\sexpansion\sof\spvcs\swithout\sAllowVolumeExpansion\sproperty$'
test/e2e/storage/testsuites/volume_expand.go:139 Nov 11 21:13:33.332: Unexpected error: <*errors.errorString | 0xc0021b07b0>: { s: "PersistentVolumeClaims [csi-hostpathxk8g2] not all in phase Bound within 5m0s", } PersistentVolumeClaims [csi-hostpathxk8g2] not all in phase Bound within 5m0s occurred test/e2e/storage/testsuites/base.go:366from junit_03.xml
[BeforeEach] [Testpattern: Dynamic PV (block volmode)] volume-expand test/e2e/storage/testsuites/base.go:93 [BeforeEach] [Testpattern: Dynamic PV (block volmode)] volume-expand test/e2e/storage/testsuites/volume_expand.go:85 [BeforeEach] [Testpattern: Dynamic PV (block volmode)] volume-expand test/e2e/framework/framework.go:151 �[1mSTEP�[0m: Creating a kubernetes client Nov 11 21:08:30.450: INFO: >>> kubeConfig: /workspace/aks147280421/kubeconfig/kubeconfig.westus2.json �[1mSTEP�[0m: Building a namespace api object, basename volume-expand �[1mSTEP�[0m: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in volume-expand-5804 �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace [It] should not allow expansion of pvcs without AllowVolumeExpansion property test/e2e/storage/testsuites/volume_expand.go:139 �[1mSTEP�[0m: deploying csi-hostpath driver Nov 11 21:08:31.131: INFO: creating *v1.ServiceAccount: volume-expand-5804/csi-attacher Nov 11 21:08:31.187: INFO: creating *v1.ClusterRole: external-attacher-runner-volume-expand-5804 Nov 11 21:08:31.187: INFO: Define cluster role external-attacher-runner-volume-expand-5804 Nov 11 21:08:31.250: INFO: creating *v1.ClusterRoleBinding: csi-attacher-role-volume-expand-5804 Nov 11 21:08:31.306: INFO: creating *v1.Role: volume-expand-5804/external-attacher-cfg-volume-expand-5804 Nov 11 21:08:31.363: INFO: creating *v1.RoleBinding: volume-expand-5804/csi-attacher-role-cfg Nov 11 21:08:31.420: INFO: creating *v1.ServiceAccount: volume-expand-5804/csi-provisioner Nov 11 21:08:31.477: INFO: creating *v1.ClusterRole: external-provisioner-runner-volume-expand-5804 Nov 11 21:08:31.477: INFO: Define cluster role external-provisioner-runner-volume-expand-5804 Nov 11 21:08:31.538: INFO: creating *v1.ClusterRoleBinding: csi-provisioner-role-volume-expand-5804 Nov 11 21:08:31.595: INFO: creating *v1.Role: volume-expand-5804/external-provisioner-cfg-volume-expand-5804 Nov 11 21:08:31.657: INFO: creating *v1.RoleBinding: volume-expand-5804/csi-provisioner-role-cfg Nov 11 21:08:31.714: INFO: creating *v1.ServiceAccount: volume-expand-5804/csi-snapshotter Nov 11 21:08:31.770: INFO: creating *v1.ClusterRole: external-snapshotter-runner-volume-expand-5804 Nov 11 21:08:31.770: INFO: Define cluster role external-snapshotter-runner-volume-expand-5804 Nov 11 21:08:31.835: INFO: creating *v1.ClusterRoleBinding: csi-snapshotter-role-volume-expand-5804 Nov 11 21:08:31.893: INFO: creating *v1.Role: volume-expand-5804/external-snapshotter-leaderelection-volume-expand-5804 Nov 11 21:08:31.949: INFO: creating *v1.RoleBinding: volume-expand-5804/external-snapshotter-leaderelection Nov 11 21:08:32.012: INFO: creating *v1.ServiceAccount: volume-expand-5804/csi-resizer Nov 11 21:08:32.067: INFO: creating *v1.ClusterRole: external-resizer-runner-volume-expand-5804 Nov 11 21:08:32.067: INFO: Define cluster role external-resizer-runner-volume-expand-5804 Nov 11 21:08:32.131: INFO: creating *v1.ClusterRoleBinding: csi-resizer-role-volume-expand-5804 Nov 11 21:08:32.191: INFO: creating *v1.Role: volume-expand-5804/external-resizer-cfg-volume-expand-5804 Nov 11 21:08:32.249: INFO: creating *v1.RoleBinding: volume-expand-5804/csi-resizer-role-cfg Nov 11 21:08:32.305: INFO: creating *v1.Service: volume-expand-5804/csi-hostpath-attacher Nov 11 21:08:32.383: INFO: creating *v1.StatefulSet: volume-expand-5804/csi-hostpath-attacher Nov 11 21:08:32.440: INFO: creating *v1beta1.CSIDriver: csi-hostpath-volume-expand-5804 Nov 11 21:08:32.526: INFO: creating *v1.Service: volume-expand-5804/csi-hostpathplugin Nov 11 21:08:32.595: INFO: creating *v1.StatefulSet: volume-expand-5804/csi-hostpathplugin Nov 11 21:08:32.656: INFO: creating *v1.Service: volume-expand-5804/csi-hostpath-provisioner Nov 11 21:08:32.733: INFO: creating *v1.StatefulSet: volume-expand-5804/csi-hostpath-provisioner Nov 11 21:08:32.792: INFO: creating *v1.Service: volume-expand-5804/csi-hostpath-resizer Nov 11 21:08:32.862: INFO: creating *v1.StatefulSet: volume-expand-5804/csi-hostpath-resizer Nov 11 21:08:32.921: INFO: creating *v1.Service: volume-expand-5804/csi-snapshotter Nov 11 21:08:32.993: INFO: creating *v1.StatefulSet: volume-expand-5804/csi-snapshotter Nov 11 21:08:33.054: INFO: creating *v1.ClusterRoleBinding: psp-csi-hostpath-role-volume-expand-5804 Nov 11 21:08:33.111: INFO: Test running for native CSI Driver, not checking metrics Nov 11 21:08:33.111: INFO: Creating resource for dynamic PV �[1mSTEP�[0m: creating a StorageClass volume-expand-5804-csi-hostpath-volume-expand-5804-sc8lbbg �[1mSTEP�[0m: creating a claim Nov 11 21:08:33.230: INFO: Waiting up to 5m0s for PersistentVolumeClaims [csi-hostpathxk8g2] to have phase Bound Nov 11 21:08:33.283: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:08:35.337: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:08:37.391: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:08:39.445: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:08:41.503: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:08:43.557: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:08:45.610: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:08:47.663: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:08:49.716: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:08:51.769: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:08:53.830: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:08:55.884: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:08:57.937: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:08:59.990: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:09:02.043: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:09:04.097: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:09:06.206: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:09:08.259: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:09:10.311: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:09:12.365: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:09:14.420: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:09:16.473: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:09:18.526: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:09:20.589: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:09:22.644: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:09:24.697: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:09:26.749: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:09:28.807: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:09:30.859: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:09:32.912: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:09:34.964: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:09:37.020: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:09:39.075: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:09:41.127: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:09:43.181: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:09:45.238: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:09:47.292: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:09:49.359: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:09:51.413: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:09:53.472: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:09:55.524: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:09:57.583: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:09:59.643: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:10:01.701: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:10:03.756: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:10:05.809: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:10:07.862: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:10:09.914: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:10:11.968: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:10:14.029: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:10:16.083: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:10:18.163: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:10:20.232: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:10:22.286: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:10:24.345: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:10:26.398: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:10:28.451: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:10:30.503: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:10:32.556: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:10:34.608: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:10:36.661: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:10:38.715: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:10:40.767: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:10:42.819: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:10:44.871: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:10:46.923: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:10:48.976: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:10:51.031: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:10:53.086: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:10:55.150: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:10:57.203: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:10:59.256: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:11:01.308: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:11:03.361: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:11:05.414: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:11:07.467: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:11:09.519: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:11:11.571: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:11:13.625: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:11:15.677: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:11:17.729: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:11:19.782: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:11:21.835: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:11:23.893: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:11:25.947: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:11:27.999: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:11:30.052: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:11:32.107: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:11:34.187: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:11:36.240: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:11:38.325: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:11:40.378: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:11:42.431: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:11:44.484: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:11:46.536: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:11:48.602: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:11:50.654: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:11:52.706: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:11:54.759: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:11:56.813: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:11:58.865: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:12:00.917: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:12:02.978: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:12:05.030: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:12:07.082: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:12:09.139: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:12:11.194: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:12:13.246: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:12:15.298: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:12:17.351: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:12:19.404: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:12:21.456: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:12:23.510: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:12:25.568: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:12:27.620: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:12:29.673: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:12:31.727: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:12:33.779: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:12:35.832: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:12:37.885: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:12:39.937: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:12:41.990: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:12:44.042: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:12:46.144: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:12:48.198: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:12:50.252: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:12:52.307: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:12:54.360: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:12:56.413: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:12:58.470: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:13:00.523: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:13:02.576: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:13:04.629: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:13:06.681: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:13:08.733: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:13:10.786: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:13:12.838: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:13:14.891: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:13:16.952: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:13:19.006: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:13:21.065: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:13:23.117: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:13:25.170: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:13:27.224: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:13:29.276: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:13:31.331: INFO: PersistentVolumeClaim csi-hostpathxk8g2 found but phase is Pending instead of Bound. Nov 11 21:13:33.332: FAIL: Unexpected error: <*errors.errorString | 0xc0021b07b0>: { s: "PersistentVolumeClaims [csi-hostpathxk8g2] not all in phase Bound within 5m0s", } PersistentVolumeClaims [csi-hostpathxk8g2] not all in phase Bound within 5m0s occurred [AfterEach] [Testpattern: Dynamic PV (block volmode)] volume-expand test/e2e/framework/framework.go:152 �[1mSTEP�[0m: Collecting events from namespace "volume-expand-5804". �[1mSTEP�[0m: Found 68 events. Nov 11 21:13:33.434: INFO: At 2019-11-11 21:08:32 +0000 UTC - event for csi-hostpath-attacher: {statefulset-controller } SuccessfulCreate: create Pod csi-hostpath-attacher-0 in StatefulSet csi-hostpath-attacher successful Nov 11 21:13:33.434: INFO: At 2019-11-11 21:08:32 +0000 UTC - event for csi-hostpath-provisioner: {statefulset-controller } SuccessfulCreate: create Pod csi-hostpath-provisioner-0 in StatefulSet csi-hostpath-provisioner successful Nov 11 21:13:33.434: INFO: At 2019-11-11 21:08:32 +0000 UTC - event for csi-hostpath-resizer: {statefulset-controller } SuccessfulCreate: create Pod csi-hostpath-resizer-0 in StatefulSet csi-hostpath-resizer successful Nov 11 21:13:33.434: INFO: At 2019-11-11 21:08:32 +0000 UTC - event for csi-hostpathplugin: {statefulset-controller } SuccessfulCreate: create Pod csi-hostpathplugin-0 in StatefulSet csi-hostpathplugin successful Nov 11 21:13:33.434: INFO: At 2019-11-11 21:08:33 +0000 UTC - event for csi-hostpathxk8g2: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "csi-hostpath-volume-expand-5804" or manually created by system administrator Nov 11 21:13:33.434: INFO: At 2019-11-11 21:08:33 +0000 UTC - event for csi-snapshotter: {statefulset-controller } SuccessfulCreate: create Pod csi-snapshotter-0 in StatefulSet csi-snapshotter successful Nov 11 21:13:33.434: INFO: At 2019-11-11 21:08:36 +0000 UTC - event for csi-hostpathplugin: {statefulset-controller } SuccessfulDelete: delete Pod csi-hostpathplugin-0 in StatefulSet csi-hostpathplugin successful Nov 11 21:13:33.434: INFO: At 2019-11-11 21:08:36 +0000 UTC - event for csi-hostpathplugin: {statefulset-controller } RecreatingFailedPod: StatefulSet volume-expand-5804/csi-hostpathplugin is recreating failed Pod csi-hostpathplugin-0 Nov 11 21:13:33.434: INFO: At 2019-11-11 21:08:36 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:33.434: INFO: At 2019-11-11 21:08:43 +0000 UTC - event for csi-hostpathplugin: {statefulset-controller } FailedCreate: create Pod csi-hostpathplugin-0 in StatefulSet csi-hostpathplugin failed error: The POST operation against Pod could not be completed at this time, please try again. Nov 11 21:13:33.434: INFO: At 2019-11-11 21:08:43 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:33.434: INFO: At 2019-11-11 21:08:46 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:33.434: INFO: At 2019-11-11 21:08:52 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:33.434: INFO: At 2019-11-11 21:08:53 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:33.434: INFO: At 2019-11-11 21:08:56 +0000 UTC - event for csi-hostpath-attacher-0: {kubelet k8s-agentpool-35271922-vmss000000} Pulling: Pulling image "quay.io/k8scsi/csi-attacher:v1.2.0" Nov 11 21:13:33.434: INFO: At 2019-11-11 21:08:57 +0000 UTC - event for csi-hostpath-provisioner-0: {kubelet k8s-agentpool-35271922-vmss000000} Pulling: Pulling image "quay.io/k8scsi/csi-provisioner:v1.4.0-rc1" Nov 11 21:13:33.434: INFO: At 2019-11-11 21:08:58 +0000 UTC - event for csi-snapshotter-0: {kubelet k8s-agentpool-35271922-vmss000000} Pulling: Pulling image "quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1" Nov 11 21:13:33.434: INFO: At 2019-11-11 21:09:00 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:33.434: INFO: At 2019-11-11 21:09:01 +0000 UTC - event for csi-hostpath-attacher-0: {kubelet k8s-agentpool-35271922-vmss000000} Pulled: Successfully pulled image "quay.io/k8scsi/csi-attacher:v1.2.0" Nov 11 21:13:33.434: INFO: At 2019-11-11 21:09:01 +0000 UTC - event for csi-hostpath-resizer-0: {kubelet k8s-agentpool-35271922-vmss000000} Pulling: Pulling image "quay.io/k8scsi/csi-resizer:v0.2.0" Nov 11 21:13:33.434: INFO: At 2019-11-11 21:09:05 +0000 UTC - event for csi-hostpath-provisioner-0: {kubelet k8s-agentpool-35271922-vmss000000} Pulled: Successfully pulled image "quay.io/k8scsi/csi-provisioner:v1.4.0-rc1" Nov 11 21:13:33.434: INFO: At 2019-11-11 21:09:05 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:33.434: INFO: At 2019-11-11 21:09:06 +0000 UTC - event for csi-hostpath-attacher-0: {kubelet k8s-agentpool-35271922-vmss000000} Created: Created container csi-attacher Nov 11 21:13:33.434: INFO: At 2019-11-11 21:09:10 +0000 UTC - event for csi-snapshotter-0: {kubelet k8s-agentpool-35271922-vmss000000} Pulled: Successfully pulled image "quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1" Nov 11 21:13:33.434: INFO: At 2019-11-11 21:09:11 +0000 UTC - event for csi-hostpath-attacher-0: {kubelet k8s-agentpool-35271922-vmss000000} Started: Started container csi-attacher Nov 11 21:13:33.434: INFO: At 2019-11-11 21:09:11 +0000 UTC - event for csi-hostpath-provisioner-0: {kubelet k8s-agentpool-35271922-vmss000000} Created: Created container csi-provisioner Nov 11 21:13:33.434: INFO: At 2019-11-11 21:09:11 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:33.434: INFO: At 2019-11-11 21:09:12 +0000 UTC - event for csi-hostpath-resizer-0: {kubelet k8s-agentpool-35271922-vmss000000} Pulled: Successfully pulled image "quay.io/k8scsi/csi-resizer:v0.2.0" Nov 11 21:13:33.434: INFO: At 2019-11-11 21:09:14 +0000 UTC - event for csi-hostpath-provisioner-0: {kubelet k8s-agentpool-35271922-vmss000000} Started: Started container csi-provisioner Nov 11 21:13:33.434: INFO: At 2019-11-11 21:09:16 +0000 UTC - event for csi-snapshotter-0: {kubelet k8s-agentpool-35271922-vmss000000} Created: Created container csi-snapshotter Nov 11 21:13:33.434: INFO: At 2019-11-11 21:09:18 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:33.434: INFO: At 2019-11-11 21:09:20 +0000 UTC - event for csi-hostpath-resizer-0: {kubelet k8s-agentpool-35271922-vmss000000} Created: Created container csi-resizer Nov 11 21:13:33.434: INFO: At 2019-11-11 21:09:20 +0000 UTC - event for csi-snapshotter-0: {kubelet k8s-agentpool-35271922-vmss000000} Started: Started container csi-snapshotter Nov 11 21:13:33.434: INFO: At 2019-11-11 21:09:22 +0000 UTC - event for csi-hostpath-resizer-0: {kubelet k8s-agentpool-35271922-vmss000000} Started: Started container csi-resizer Nov 11 21:13:33.434: INFO: At 2019-11-11 21:09:25 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:33.434: INFO: At 2019-11-11 21:09:28 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:33.434: INFO: At 2019-11-11 21:09:35 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:33.434: INFO: At 2019-11-11 21:09:39 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:33.434: INFO: At 2019-11-11 21:09:42 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:33.434: INFO: At 2019-11-11 21:09:43 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:33.434: INFO: At 2019-11-11 21:09:48 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:33.434: INFO: At 2019-11-11 21:10:03 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:33.434: INFO: At 2019-11-11 21:10:11 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:33.434: INFO: At 2019-11-11 21:10:19 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:33.434: INFO: At 2019-11-11 21:10:24 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:33.434: INFO: At 2019-11-11 21:10:32 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:33.434: INFO: At 2019-11-11 21:10:38 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:33.434: INFO: At 2019-11-11 21:10:47 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:33.434: INFO: At 2019-11-11 21:10:55 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:33.434: INFO: At 2019-11-11 21:11:01 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:33.434: INFO: At 2019-11-11 21:11:09 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:33.434: INFO: At 2019-11-11 21:11:19 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:33.434: INFO: At 2019-11-11 21:11:26 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:33.434: INFO: At 2019-11-11 21:11:38 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:33.434: INFO: At 2019-11-11 21:11:51 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:33.434: INFO: At 2019-11-11 21:12:05 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:33.435: INFO: At 2019-11-11 21:12:14 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:33.435: INFO: At 2019-11-11 21:12:22 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:33.435: INFO: At 2019-11-11 21:12:30 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:33.435: INFO: At 2019-11-11 21:12:39 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:33.435: INFO: At 2019-11-11 21:12:48 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:33.435: INFO: At 2019-11-11 21:12:54 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:33.435: INFO: At 2019-11-11 21:12:58 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:33.435: INFO: At 2019-11-11 21:13:00 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:33.435: INFO: At 2019-11-11 21:13:03 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:33.435: INFO: At 2019-11-11 21:13:06 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:33.435: INFO: At 2019-11-11 21:13:17 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:33.435: INFO: At 2019-11-11 21:13:30 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:33.488: INFO: POD NODE PHASE GRACE CONDITIONS Nov 11 21:13:33.488: INFO: csi-hostpath-attacher-0 k8s-agentpool-35271922-vmss000000 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:08:34 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:09:20 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:09:20 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:08:34 +0000 UTC }] Nov 11 21:13:33.488: INFO: csi-hostpath-provisioner-0 k8s-agentpool-35271922-vmss000000 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:08:36 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:09:25 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:09:25 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:08:36 +0000 UTC }] Nov 11 21:13:33.488: INFO: csi-hostpath-resizer-0 k8s-agentpool-35271922-vmss000000 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:08:37 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:09:28 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:09:28 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:08:37 +0000 UTC }] Nov 11 21:13:33.488: INFO: csi-hostpathplugin-0 k8s-agentpool-35271922-vmss000000 Pending [] Nov 11 21:13:33.488: INFO: csi-snapshotter-0 k8s-agentpool-35271922-vmss000000 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:08:38 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:09:20 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:09:20 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:08:38 +0000 UTC }] Nov 11 21:13:33.488: INFO: Nov 11 21:13:33.593: INFO: Logging node info for node k8s-agentpool-35271922-vmss000000 Nov 11 21:13:33.645: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool-35271922-vmss000000 /api/v1/nodes/k8s-agentpool-35271922-vmss000000 04da751e-f23a-470e-b29d-e591113d7271 41796 0 2019-11-11 20:48:07 +0000 UTC <nil> <nil> map[agentpool:agentpool beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubelet_cleanup:true kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool-35271922-vmss000000 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: storageprofile:managed storagetier:Premium_LRS] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-2030":"k8s-agentpool-35271922-vmss000000","csi-hostpath-ephemeral-7436":"k8s-agentpool-35271922-vmss000000","csi-hostpath-provisioning-3597":"k8s-agentpool-35271922-vmss000000","csi-hostpath-provisioning-9334":"k8s-agentpool-35271922-vmss000000","csi-hostpath-volume-5092":"k8s-agentpool-35271922-vmss000000","csi-hostpath-volume-expand-1449":"k8s-agentpool-35271922-vmss000000","csi-mock-csi-mock-volumes-1829":"csi-mock-csi-mock-volumes-1829","csi-mock-csi-mock-volumes-2481":"csi-mock-csi-mock-volumes-2481","csi-mock-csi-mock-volumes-763":"csi-mock-csi-mock-volumes-763","csi-mock-csi-mock-volumes-9721":"csi-mock-csi-mock-volumes-9721"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-35271922-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16797044736 0} {<nil>} 16403364Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16010612736 0} {<nil>} 15635364Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:13:03 +0000 UTC,LastTransitionTime:2019-11-11 20:47:58 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:13:03 +0000 UTC,LastTransitionTime:2019-11-11 20:47:58 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:13:03 +0000 UTC,LastTransitionTime:2019-11-11 20:47:58 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:13:03 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.248.0.4,},NodeAddress{Type:Hostname,Address:k8s-agentpool-35271922-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:84c06f3ee41a47aaa558c81f34711895,SystemUUID:9BB24974-637B-EE4C-A221-0F74BB743424,BootID:59f7eb0c-c09c-4eae-a14b-0710f9e14004,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[quay.io/kubernetes_incubator/nfs-provisioner@sha256:df762117e3c891f2d2ddff46ecb0776ba1f9f3c44cfd7739b0683bcd7a7954a8 quay.io/kubernetes_incubator/nfs-provisioner:v2.2.2],SizeBytes:391772778,},ContainerImage{Names:[gluster/glusterdynamic-provisioner@sha256:90067cb05a7d217651e84576935934fd9dcff8e6bcdcbaa416bbf36fcd09dbd1 gluster/glusterdynamic-provisioner:v1.0],SizeBytes:373281573,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/volume/nfs@sha256:c2ad734346f608a5f7d69cfded93c4e8094069320657bd372d12ba21dea3ea71 gcr.io/kubernetes-e2e-test-images/volume/nfs:1.0],SizeBytes:225358913,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/jessie-dnsutils@sha256:ad583e33cb284f7ef046673809b146ec4053cda19b54a85d2b180a86169715eb gcr.io/kubernetes-e2e-test-images/jessie-dnsutils:1.0],SizeBytes:195659796,},ContainerImage{Names:[httpd@sha256:addd70e4ee83f3bc9a4c1c7c41e37927ba47faf639312fc936df3afad7926f5a httpd:2.4.39-alpine],SizeBytes:126894770,},ContainerImage{Names:[httpd@sha256:eb8ccf084cf3e80eece1add239effefd171eb39adbc154d33c14260d905d4060 httpd:2.4.38-alpine],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1],SizeBytes:121711221,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727 gcr.io/kubernetes-e2e-test-images/agnhost:2.6],SizeBytes:57345321,},ContainerImage{Names:[quay.io/k8scsi/csi-provisioner@sha256:0efcb424f1dde9b9fb11a1a14f2e48ab47e1c3f08bc3a929990dcfcb1f7ab34f quay.io/k8scsi/csi-provisioner:v1.4.0-rc1],SizeBytes:54431016,},ContainerImage{Names:[quay.io/k8scsi/csi-snapshotter@sha256:e3d3e742e32d00488fdb401045b9b1d033d7ca0ab6e760f77b24750fc95e5f70 quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1],SizeBytes:51703561,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0],SizeBytes:46226754,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:0aba670b4d9d6b2e720bbf575d733156c676b693ca26501235444490300db838 quay.io/k8scsi/csi-attacher:v1.1.0],SizeBytes:42839085,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:7d46fb6eb8b890dc546029d1565d502b4a1d974d33625c6ee2bc7991b77fc1a1 quay.io/k8scsi/csi-resizer:v0.2.0],SizeBytes:42817100,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:f315c9042e56def3c05c6b04fe79ec9da6d39ddc557ca365a76cf35964ea08b6 quay.io/k8scsi/csi-resizer:v0.1.0],SizeBytes:42623056,},ContainerImage{Names:[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0],SizeBytes:42488424,},ContainerImage{Names:[quay.io/k8scsi/hostpathplugin@sha256:b4826e492fc1762fceaf9726f41575ca0a4567864d3d235da874818de18039de quay.io/k8scsi/hostpathplugin:v1.2.0-rc5],SizeBytes:28761497,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7],SizeBytes:22909487,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/echoserver@sha256:e9ba514b896cdf559eef8788b66c2c3ee55f3572df617647b4b0d8b6bf81cf19 gcr.io/kubernetes-e2e-test-images/echoserver:2.2],SizeBytes:21692741,},ContainerImage{Names:[quay.io/k8scsi/mock-driver@sha256:e0eed916b7d970bad2b7d9875f9ad16932f987f0f3d91ec5d86da68b0b5cc9d1 quay.io/k8scsi/mock-driver:v2.1.0],SizeBytes:16226335,},ContainerImage{Names:[nginx@sha256:485b610fefec7ff6c463ced9623314a04ed67e3945b9c08d7e53a47f6d108dc7 nginx:1.14-alpine],SizeBytes:16032814,},ContainerImage{Names:[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0],SizeBytes:15815995,},ContainerImage{Names:[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0],SizeBytes:14967303,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/dnsutils@sha256:2abeee84efb79c14d731966e034af33bf324d3b26ca28497555511ff094b3ddd gcr.io/kubernetes-e2e-test-images/dnsutils:1.1],SizeBytes:9349974,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nautilus@sha256:33a732d4c42a266912a5091598a0f07653c9134db4b8d571690d8afd509e0bfc gcr.io/kubernetes-e2e-test-images/nautilus:1.0],SizeBytes:4753501,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/test-webserver@sha256:7f93d6e32798ff28bc6289254d0c2867fe2c849c8e46edc50f8624734309812e gcr.io/kubernetes-e2e-test-images/test-webserver:1.0],SizeBytes:4732240,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 gcr.io/kubernetes-e2e-test-images/mounttest:1.0],SizeBytes:1563521,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29],SizeBytes:1154361,},ContainerImage{Names:[busybox@sha256:bbc3a03235220b170ba48a157dd097dd1379299370e1ed99ce976df0355d24f0 busybox:1.27],SizeBytes:1129289,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea k8s.gcr.io/pause-amd64:3.1 k8s.gcr.io/pause:3.1],SizeBytes:742472,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-763^4],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-763^4,DevicePath:,},},Config:nil,},} Nov 11 21:13:33.646: INFO: Logging kubelet events for node k8s-agentpool-35271922-vmss000000 Nov 11 21:13:33.702: INFO: Logging pods the kubelet thinks is on node k8s-agentpool-35271922-vmss000000 Nov 11 21:13:33.822: INFO: csi-hostpathplugin-0 started at 2019-11-11 21:13:31 +0000 UTC (0+0 container statuses recorded) Nov 11 21:13:33.822: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:12:23 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:33.822: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:13:33.822: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:09:14 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:33.822: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:13:33.822: INFO: csi-hostpathplugin-0 started at <nil> (0+0 container statuses recorded) Nov 11 21:13:33.822: INFO: kubernetes-dashboard-65966766b9-lv2x8 started at 2019-11-11 20:48:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:33.822: INFO: Container kubernetes-dashboard ready: true, restart count 0 Nov 11 21:13:33.822: INFO: csi-snapshotter-0 started at 2019-11-11 21:08:38 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:33.822: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 11 21:13:33.822: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:13:12 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:33.822: INFO: Container agnhost ready: false, restart count 0 Nov 11 21:13:33.822: INFO: csi-snapshotter-0 started at 2019-11-11 21:10:29 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:33.822: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 11 21:13:33.822: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-sqdt4 started at 2019-11-11 21:09:54 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:33.822: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: false, restart count 0 Nov 11 21:13:33.822: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:07:21 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:33.822: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:13:33.822: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-7ndbp started at 2019-11-11 21:09:58 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:33.822: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: true, restart count 0 Nov 11 21:13:33.822: INFO: csi-hostpathplugin-0 started at 2019-11-11 21:04:26 +0000 UTC (0+3 container statuses recorded) Nov 11 21:13:33.822: INFO: Container hostpath ready: true, restart count 0 Nov 11 21:13:33.822: INFO: Container liveness-probe ready: true, restart count 0 Nov 11 21:13:33.822: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 11 21:13:33.822: INFO: webserver-dc8bd894f-vm4pq started at 2019-11-11 21:12:41 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:33.822: INFO: Container httpd ready: false, restart count 0 Nov 11 21:13:33.822: INFO: exec-volume-test-local-preprovisionedpv-hml6 started at 2019-11-11 21:13:24 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:33.822: INFO: Container exec-container-local-preprovisionedpv-hml6 ready: false, restart count 0 Nov 11 21:13:33.822: INFO: csi-hostpath-resizer-0 started at 2019-11-11 21:07:52 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:33.822: INFO: Container csi-resizer ready: false, restart count 0 Nov 11 21:13:33.822: INFO: external-provisioner-6rv66 started at 2019-11-11 21:11:45 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:33.822: INFO: Container nfs-provisioner ready: true, restart count 0 Nov 11 21:13:33.822: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-n58zl started at 2019-11-11 21:09:53 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:33.822: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: true, restart count 0 Nov 11 21:13:33.822: INFO: csi-hostpath-resizer-0 started at 2019-11-11 21:10:28 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:33.822: INFO: Container csi-resizer ready: true, restart count 0 Nov 11 21:13:33.822: INFO: pod-subpath-test-local-preprovisionedpv-rqnf started at 2019-11-11 21:11:54 +0000 UTC (0+2 container statuses recorded) Nov 11 21:13:33.822: INFO: Container test-container-subpath-local-preprovisionedpv-rqnf ready: false, restart count 0 Nov 11 21:13:33.822: INFO: Container test-container-volume-local-preprovisionedpv-rqnf ready: true, restart count 0 Nov 11 21:13:33.822: INFO: pod-subpath-test-local-preprovisionedpv-9cfl started at 2019-11-11 21:13:22 +0000 UTC (2+1 container statuses recorded) Nov 11 21:13:33.822: INFO: Init container init-volume-local-preprovisionedpv-9cfl ready: false, restart count 0 Nov 11 21:13:33.822: INFO: Init container test-init-subpath-local-preprovisionedpv-9cfl ready: false, restart count 0 Nov 11 21:13:33.822: INFO: Container test-container-subpath-local-preprovisionedpv-9cfl ready: false, restart count 0 Nov 11 21:13:33.822: INFO: blobfuse-flexvol-installer-dmnhw started at 2019-11-11 20:48:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:33.822: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 Nov 11 21:13:33.822: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-vdczm started at 2019-11-11 21:10:00 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:33.822: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: true, restart count 0 Nov 11 21:13:33.822: INFO: csi-hostpath-provisioner-0 started at 2019-11-11 21:02:40 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:33.822: INFO: Container csi-provisioner ready: true, restart count 0 Nov 11 21:13:33.822: INFO: coredns-87f5d796-6mz2l started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:33.822: INFO: Container coredns ready: true, restart count 0 Nov 11 21:13:33.822: INFO: csi-hostpath-attacher-0 started at 2019-11-11 21:02:37 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:33.822: INFO: Container csi-attacher ready: true, restart count 0 Nov 11 21:13:33.822: INFO: security-context-50ac59ee-3233-4a07-9d18-a85cc2c4304d started at 2019-11-11 21:12:34 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:33.822: INFO: Container write-pod ready: false, restart count 0 Nov 11 21:13:33.822: INFO: csi-hostpath-resizer-0 started at 2019-11-11 21:08:37 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:33.822: INFO: Container csi-resizer ready: true, restart count 0 Nov 11 21:13:33.822: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-695c7 started at 2019-11-11 21:09:56 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:33.822: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: true, restart count 0 Nov 11 21:13:33.822: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:11:30 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:33.823: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:13:33.823: INFO: csi-hostpath-provisioner-0 started at 2019-11-11 21:07:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:33.823: INFO: Container csi-provisioner ready: true, restart count 0 Nov 11 21:13:33.823: INFO: pod-subpath-test-local-preprovisionedpv-vhmb started at 2019-11-11 21:13:20 +0000 UTC (0+2 container statuses recorded) Nov 11 21:13:33.823: INFO: Container test-container-subpath-local-preprovisionedpv-vhmb ready: false, restart count 0 Nov 11 21:13:33.823: INFO: Container test-container-volume-local-preprovisionedpv-vhmb ready: false, restart count 0 Nov 11 21:13:33.823: INFO: csi-hostpath-attacher-0 started at 2019-11-11 21:08:34 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:33.823: INFO: Container csi-attacher ready: true, restart count 0 Nov 11 21:13:33.823: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-77l5m started at 2019-11-11 21:09:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:33.823: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: true, restart count 0 Nov 11 21:13:33.823: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-fmq2z started at 2019-11-11 21:09:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:33.823: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: true, restart count 0 Nov 11 21:13:33.823: INFO: webserver-77c9474f75-ff62s started at 2019-11-11 21:12:41 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:33.823: INFO: Container httpd ready: false, restart count 0 Nov 11 21:13:33.823: INFO: kube-proxy-nwdpz started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:33.823: INFO: Container kube-proxy ready: true, restart count 0 Nov 11 21:13:33.823: INFO: csi-snapshotter-0 started at 2019-11-11 21:07:53 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:33.823: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 11 21:13:33.823: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-6wrg6 started at 2019-11-11 21:09:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:33.823: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: false, restart count 0 Nov 11 21:13:33.823: INFO: local-client started at 2019-11-11 21:10:53 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:33.823: INFO: Container local-client ready: true, restart count 0 Nov 11 21:13:33.823: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:11:48 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:33.823: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:13:33.823: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-h7bv7 started at 2019-11-11 21:10:01 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:33.823: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: true, restart count 0 Nov 11 21:13:33.823: INFO: csi-mockplugin-attacher-0 started at 2019-11-11 21:11:13 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:33.823: INFO: Container csi-attacher ready: true, restart count 0 Nov 11 21:13:33.823: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:10:12 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:33.823: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:13:33.823: INFO: azure-ip-masq-agent-c4mb2 started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:33.823: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:13:33.823: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-rlsfg started at 2019-11-11 21:09:55 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:33.823: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: true, restart count 0 Nov 11 21:13:33.823: INFO: pvc-volume-tester-894sb started at 2019-11-11 21:12:38 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:33.823: INFO: Container volume-tester ready: false, restart count 0 Nov 11 21:13:33.823: INFO: csi-hostpath-provisioner-0 started at 2019-11-11 21:08:36 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:33.823: INFO: Container csi-provisioner ready: true, restart count 0 Nov 11 21:13:33.823: INFO: csi-hostpath-provisioner-0 started at 2019-11-11 21:10:26 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:33.823: INFO: Container csi-provisioner ready: true, restart count 0 Nov 11 21:13:33.823: INFO: ss2-0 started at 2019-11-11 21:10:44 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:33.823: INFO: Container webserver ready: true, restart count 0 Nov 11 21:13:33.823: INFO: csi-mockplugin-0 started at 2019-11-11 21:11:12 +0000 UTC (0+3 container statuses recorded) Nov 11 21:13:33.823: INFO: Container csi-provisioner ready: true, restart count 0 Nov 11 21:13:33.823: INFO: Container driver-registrar ready: true, restart count 0 Nov 11 21:13:33.823: INFO: Container mock ready: true, restart count 0 Nov 11 21:13:33.823: INFO: csi-mockplugin-resizer-0 started at 2019-11-11 21:11:14 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:33.823: INFO: Container csi-resizer ready: true, restart count 0 Nov 11 21:13:33.823: INFO: keyvault-flexvolume-zzp7p started at 2019-11-11 20:48:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:33.823: INFO: Container keyvault-flexvolume ready: true, restart count 0 Nov 11 21:13:33.823: INFO: csi-hostpath-attacher-0 started at 2019-11-11 21:07:46 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:33.823: INFO: Container csi-attacher ready: true, restart count 0 Nov 11 21:13:33.823: INFO: csi-snapshotter-0 started at 2019-11-11 21:02:41 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:33.823: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 11 21:13:33.823: INFO: csi-hostpath-attacher-0 started at 2019-11-11 21:10:24 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:33.823: INFO: Container csi-attacher ready: true, restart count 0 Nov 11 21:13:33.823: INFO: rs-cnbnm started at 2019-11-11 21:10:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:33.823: INFO: Container busybox ready: true, restart count 0 Nov 11 21:13:33.823: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:11:34 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:33.823: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:13:33.823: INFO: csi-hostpath-resizer-0 started at 2019-11-11 21:02:40 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:33.823: INFO: Container csi-resizer ready: true, restart count 0 W1111 21:13:33.877961 93386 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:13:34.778: INFO: Latency metrics for node k8s-agentpool-35271922-vmss000000 Nov 11 21:13:34.778: INFO: Logging node info for node k8s-agentpool-35271922-vmss000001 Nov 11 21:13:34.830: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool-35271922-vmss000001 /api/v1/nodes/k8s-agentpool-35271922-vmss000001 af5ba0b6-4f7f-4468-9d3c-b3e116ca81ca 41314 0 2019-11-11 20:48:13 +0000 UTC <nil> <nil> map[agentpool:agentpool beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubelet_cleanup:true kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool-35271922-vmss000001 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: storageprofile:managed storagetier:Premium_LRS] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-8406":"k8s-agentpool-35271922-vmss000001","csi-hostpath-provisioning-2466":"k8s-agentpool-35271922-vmss000001","csi-hostpath-provisioning-265":"k8s-agentpool-35271922-vmss000001","csi-hostpath-provisioning-310":"k8s-agentpool-35271922-vmss000001","csi-hostpath-volume-expand-3983":"k8s-agentpool-35271922-vmss000001","csi-mock-csi-mock-volumes-2781":"csi-mock-csi-mock-volumes-2781","csi-mock-csi-mock-volumes-2838":"csi-mock-csi-mock-volumes-2838","csi-mock-csi-mock-volumes-3306":"csi-mock-csi-mock-volumes-3306","csi-mock-csi-mock-volumes-6769":"csi-mock-csi-mock-volumes-6769","csi-mock-csi-mock-volumes-694":"csi-mock-csi-mock-volumes-694","csi-mock-csi-mock-volumes-797":"csi-mock-csi-mock-volumes-797"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-35271922-vmss/virtualMachines/1,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16797044736 0} {<nil>} 16403364Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16010612736 0} {<nil>} 15635364Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:12:38 +0000 UTC,LastTransitionTime:2019-11-11 20:48:13 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:12:38 +0000 UTC,LastTransitionTime:2019-11-11 20:48:13 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:12:38 +0000 UTC,LastTransitionTime:2019-11-11 20:48:13 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:12:38 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.248.0.5,},NodeAddress{Type:Hostname,Address:k8s-agentpool-35271922-vmss000001,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:1df8c22200b84e64b76150d232ecb929,SystemUUID:B03EE801-7A2D-7140-BE20-B2C844575A73,BootID:d32206f7-48b0-46c5-a542-39721566b456,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[quay.io/kubernetes_incubator/nfs-provisioner@sha256:df762117e3c891f2d2ddff46ecb0776ba1f9f3c44cfd7739b0683bcd7a7954a8 quay.io/kubernetes_incubator/nfs-provisioner:v2.2.2],SizeBytes:391772778,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/volume/nfs@sha256:c2ad734346f608a5f7d69cfded93c4e8094069320657bd372d12ba21dea3ea71 gcr.io/kubernetes-e2e-test-images/volume/nfs:1.0],SizeBytes:225358913,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/jessie-dnsutils@sha256:ad583e33cb284f7ef046673809b146ec4053cda19b54a85d2b180a86169715eb gcr.io/kubernetes-e2e-test-images/jessie-dnsutils:1.0],SizeBytes:195659796,},ContainerImage{Names:[httpd@sha256:addd70e4ee83f3bc9a4c1c7c41e37927ba47faf639312fc936df3afad7926f5a httpd:2.4.39-alpine],SizeBytes:126894770,},ContainerImage{Names:[httpd@sha256:eb8ccf084cf3e80eece1add239effefd171eb39adbc154d33c14260d905d4060 httpd:2.4.38-alpine],SizeBytes:123781643,},ContainerImage{Names:[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0],SizeBytes:71821984,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727 gcr.io/kubernetes-e2e-test-images/agnhost:2.6],SizeBytes:57345321,},ContainerImage{Names:[quay.io/k8scsi/csi-provisioner@sha256:0efcb424f1dde9b9fb11a1a14f2e48ab47e1c3f08bc3a929990dcfcb1f7ab34f quay.io/k8scsi/csi-provisioner:v1.4.0-rc1],SizeBytes:54431016,},ContainerImage{Names:[quay.io/k8scsi/csi-snapshotter@sha256:e3d3e742e32d00488fdb401045b9b1d033d7ca0ab6e760f77b24750fc95e5f70 quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1],SizeBytes:51703561,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0],SizeBytes:46226754,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:0aba670b4d9d6b2e720bbf575d733156c676b693ca26501235444490300db838 quay.io/k8scsi/csi-attacher:v1.1.0],SizeBytes:42839085,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:7d46fb6eb8b890dc546029d1565d502b4a1d974d33625c6ee2bc7991b77fc1a1 quay.io/k8scsi/csi-resizer:v0.2.0],SizeBytes:42817100,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:f315c9042e56def3c05c6b04fe79ec9da6d39ddc557ca365a76cf35964ea08b6 quay.io/k8scsi/csi-resizer:v0.1.0],SizeBytes:42623056,},ContainerImage{Names:[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1],SizeBytes:42541759,},ContainerImage{Names:[gcr.io/google-containers/debian-base@sha256:6966a0aedd7592c18ff2dd803c08bd85780ee19f5e3a2e7cf908a4cd837afcde gcr.io/google-containers/debian-base:0.4.1],SizeBytes:42323657,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nonroot@sha256:d4ede5c74517090b6686219059118ed178cf4620f5db8781b32f806bb1e7395b gcr.io/kubernetes-e2e-test-images/nonroot:1.0],SizeBytes:42321438,},ContainerImage{Names:[redis@sha256:50899ea1ceed33fa03232f3ac57578a424faa1742c1ac9c7a7bdb95cdf19b858 redis:5.0.5-alpine],SizeBytes:29331594,},ContainerImage{Names:[quay.io/k8scsi/hostpathplugin@sha256:b4826e492fc1762fceaf9726f41575ca0a4567864d3d235da874818de18039de quay.io/k8scsi/hostpathplugin:v1.2.0-rc5],SizeBytes:28761497,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7],SizeBytes:22909487,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/echoserver@sha256:e9ba514b896cdf559eef8788b66c2c3ee55f3572df617647b4b0d8b6bf81cf19 gcr.io/kubernetes-e2e-test-images/echoserver:2.2],SizeBytes:21692741,},ContainerImage{Names:[quay.io/k8scsi/mock-driver@sha256:e0eed916b7d970bad2b7d9875f9ad16932f987f0f3d91ec5d86da68b0b5cc9d1 quay.io/k8scsi/mock-driver:v2.1.0],SizeBytes:16226335,},ContainerImage{Names:[nginx@sha256:485b610fefec7ff6c463ced9623314a04ed67e3945b9c08d7e53a47f6d108dc7 nginx:1.14-alpine],SizeBytes:16032814,},ContainerImage{Names:[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0],SizeBytes:15815995,},ContainerImage{Names:[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0],SizeBytes:14967303,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/dnsutils@sha256:2abeee84efb79c14d731966e034af33bf324d3b26ca28497555511ff094b3ddd gcr.io/kubernetes-e2e-test-images/dnsutils:1.1],SizeBytes:9349974,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nonewprivs@sha256:10066e9039219449fe3c81f38fe01928f87914150768ab81b62a468e51fa7411 gcr.io/kubernetes-e2e-test-images/nonewprivs:1.0],SizeBytes:6757579,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nautilus@sha256:33a732d4c42a266912a5091598a0f07653c9134db4b8d571690d8afd509e0bfc gcr.io/kubernetes-e2e-test-images/nautilus:1.0],SizeBytes:4753501,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/test-webserver@sha256:7f93d6e32798ff28bc6289254d0c2867fe2c849c8e46edc50f8624734309812e gcr.io/kubernetes-e2e-test-images/test-webserver:1.0],SizeBytes:4732240,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:4206620,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 gcr.io/kubernetes-e2e-test-images/mounttest:1.0],SizeBytes:1563521,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest-user@sha256:17319ca525ee003681fccf7e8c6b1b910ff4f49b653d939ac7f9b6e7c463933d gcr.io/kubernetes-e2e-test-images/mounttest-user:1.0],SizeBytes:1450451,},ContainerImage{Names:[busybox@sha256:1303dbf110c57f3edf68d9f5a16c082ec06c4cf7604831669faf2c712260b5a0 busybox:latest],SizeBytes:1219790,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29],SizeBytes:1154361,},ContainerImage{Names:[busybox@sha256:bbc3a03235220b170ba48a157dd097dd1379299370e1ed99ce976df0355d24f0 busybox:1.27],SizeBytes:1129289,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea k8s.gcr.io/pause-amd64:3.1 k8s.gcr.io/pause:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:13:34.831: INFO: Logging kubelet events for node k8s-agentpool-35271922-vmss000001 Nov 11 21:13:34.887: INFO: Logging pods the kubelet thinks is on node k8s-agentpool-35271922-vmss000001 Nov 11 21:13:35.016: INFO: rs-k269h started at 2019-11-11 21:10:53 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:35.016: INFO: Container busybox ready: true, restart count 0 Nov 11 21:13:35.016: INFO: webserver-dc8bd894f-ldzf4 started at 2019-11-11 21:12:36 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:35.016: INFO: Container httpd ready: false, restart count 0 Nov 11 21:13:35.016: INFO: webserver-6f55b98-6gv2w started at 2019-11-11 21:12:25 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:35.016: INFO: Container httpd ready: true, restart count 0 Nov 11 21:13:35.016: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-gm5b8 started at 2019-11-11 21:10:00 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:35.016: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: false, restart count 0 Nov 11 21:13:35.016: INFO: keyvault-flexvolume-2s2wq started at 2019-11-11 20:48:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:35.016: INFO: Container keyvault-flexvolume ready: true, restart count 0 Nov 11 21:13:35.016: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-sn584 started at 2019-11-11 21:09:53 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:35.016: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: false, restart count 0 Nov 11 21:13:35.016: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-nb9gn started at 2019-11-11 21:09:52 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:35.016: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: false, restart count 0 Nov 11 21:13:35.016: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-s2r9q started at 2019-11-11 21:10:02 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:35.016: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: false, restart count 0 Nov 11 21:13:35.016: INFO: webserver-dc8bd894f-gwxx7 started at 2019-11-11 21:12:35 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:35.016: INFO: Container httpd ready: false, restart count 0 Nov 11 21:13:35.016: INFO: webserver-77c9474f75-vmx9l started at 2019-11-11 21:12:38 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:35.016: INFO: Container httpd ready: false, restart count 0 Nov 11 21:13:35.016: INFO: pod-configmaps-3095fd89-26c2-4c91-bc11-f89acc95a563 started at 2019-11-11 21:12:09 +0000 UTC (0+3 container statuses recorded) Nov 11 21:13:35.016: INFO: Container createcm-volume-test ready: true, restart count 0 Nov 11 21:13:35.016: INFO: Container delcm-volume-test ready: true, restart count 0 Nov 11 21:13:35.016: INFO: Container updcm-volume-test ready: true, restart count 0 Nov 11 21:13:35.016: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-v8m7z started at 2019-11-11 21:09:50 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:35.016: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: false, restart count 0 Nov 11 21:13:35.016: INFO: ss2-2 started at 2019-11-11 21:06:58 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:35.016: INFO: Container webserver ready: false, restart count 0 Nov 11 21:13:35.016: INFO: rs-gkz92 started at 2019-11-11 21:10:52 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:35.016: INFO: Container busybox ready: true, restart count 0 Nov 11 21:13:35.016: INFO: webserver-6f55b98-kphlw started at 2019-11-11 21:12:18 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:35.016: INFO: Container httpd ready: true, restart count 0 Nov 11 21:13:35.016: INFO: kube-proxy-sgvk2 started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:35.016: INFO: Container kube-proxy ready: true, restart count 0 Nov 11 21:13:35.016: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-xvlpv started at 2019-11-11 21:09:55 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:35.016: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: false, restart count 0 Nov 11 21:13:35.016: INFO: csi-mockplugin-0 started at 2019-11-11 21:07:06 +0000 UTC (0+3 container statuses recorded) Nov 11 21:13:35.016: INFO: Container csi-provisioner ready: false, restart count 0 Nov 11 21:13:35.016: INFO: Container driver-registrar ready: false, restart count 0 Nov 11 21:13:35.016: INFO: Container mock ready: false, restart count 0 Nov 11 21:13:35.016: INFO: azure-ip-masq-agent-9427k started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:35.016: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:13:35.016: INFO: hostexec-k8s-agentpool-35271922-vmss000001 started at 2019-11-11 21:12:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:35.016: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:13:35.016: INFO: var-expansion-cd36be16-75e0-4d5c-a5fc-6f731ce949bb started at 2019-11-11 21:12:55 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:35.016: INFO: Container dapi-container ready: false, restart count 0 Nov 11 21:13:35.016: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-5xrt6 started at 2019-11-11 21:09:49 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:35.016: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: false, restart count 0 Nov 11 21:13:35.016: INFO: webserver-77c9474f75-b66lx started at 2019-11-11 21:12:43 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:35.016: INFO: Container httpd ready: false, restart count 0 Nov 11 21:13:35.016: INFO: blobfuse-flexvol-installer-tvkwl started at 2019-11-11 20:48:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:35.016: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 Nov 11 21:13:35.016: INFO: tiller-deploy-7559b6b885-jsnn2 started at 2019-11-11 20:48:53 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:35.016: INFO: Container tiller ready: true, restart count 0 Nov 11 21:13:35.016: INFO: ss2-1 started at 2019-11-11 21:08:52 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:35.016: INFO: Container webserver ready: true, restart count 0 Nov 11 21:13:35.016: INFO: downwardapi-volume-d6793c9e-82b1-459d-900a-4432db1f6b18 started at 2019-11-11 21:13:25 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:35.016: INFO: Container client-container ready: false, restart count 0 Nov 11 21:13:35.016: INFO: webserver-dc8bd894f-5pfhd started at 2019-11-11 21:12:43 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:35.016: INFO: Container httpd ready: false, restart count 0 Nov 11 21:13:35.016: INFO: pod-subpath-test-nfs-dynamicpv-rjm7 started at 2019-11-11 21:13:09 +0000 UTC (2+1 container statuses recorded) Nov 11 21:13:35.016: INFO: Init container init-volume-nfs-dynamicpv-rjm7 ready: false, restart count 0 Nov 11 21:13:35.016: INFO: Init container test-init-subpath-nfs-dynamicpv-rjm7 ready: false, restart count 0 Nov 11 21:13:35.016: INFO: Container test-container-subpath-nfs-dynamicpv-rjm7 ready: false, restart count 0 Nov 11 21:13:35.016: INFO: webserver-77c9474f75-zws9d started at 2019-11-11 21:12:37 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:35.016: INFO: Container httpd ready: false, restart count 0 Nov 11 21:13:35.016: INFO: metrics-server-58ff8c5ddf-thxvp started at 2019-11-11 20:48:53 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:35.016: INFO: Container metrics-server ready: true, restart count 0 Nov 11 21:13:35.016: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-vzxn8 started at 2019-11-11 21:09:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:35.016: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: false, restart count 0 Nov 11 21:13:35.016: INFO: replace-1573506660-vrzrd started at 2019-11-11 21:11:09 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:35.016: INFO: Container c ready: true, restart count 0 Nov 11 21:13:35.016: INFO: csi-mockplugin-attacher-0 started at 2019-11-11 21:07:07 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:35.016: INFO: Container csi-attacher ready: false, restart count 0 Nov 11 21:13:35.016: INFO: external-provisioner-64xss started at 2019-11-11 21:12:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:35.016: INFO: Container nfs-provisioner ready: false, restart count 0 Nov 11 21:13:35.016: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-m44k9 started at 2019-11-11 21:09:58 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:35.016: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: false, restart count 0 Nov 11 21:13:35.016: INFO: pod-ready started at 2019-11-11 21:10:19 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:35.016: INFO: Container pod-readiness-gate ready: false, restart count 0 Nov 11 21:13:35.016: INFO: pod-prestop-hook-b9c29d46-a5ed-430d-beb3-947a38845e94 started at 2019-11-11 21:12:11 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:35.016: INFO: Container nginx ready: true, restart count 0 Nov 11 21:13:35.016: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-cbdrk started at 2019-11-11 21:09:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:35.016: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: false, restart count 0 W1111 21:13:35.077440 93386 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:13:35.451: INFO: Latency metrics for node k8s-agentpool-35271922-vmss000001 Nov 11 21:13:35.451: INFO: Logging node info for node k8s-master-35271922-vmss000000 Nov 11 21:13:35.505: INFO: Node Info: &Node{ObjectMeta:{k8s-master-35271922-vmss000000 /api/v1/nodes/k8s-master-35271922-vmss000000 f042d5b7-5ee8-4528-ae46-c997fef88c1c 41778 0 2019-11-11 20:48:07 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-35271922-vmss000000 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-35271922-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284363264 0} {<nil>} 7113636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6497931264 0} {<nil>} 6345636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:13:01 +0000 UTC,LastTransitionTime:2019-11-11 20:48:04 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:13:01 +0000 UTC,LastTransitionTime:2019-11-11 20:48:04 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:13:01 +0000 UTC,LastTransitionTime:2019-11-11 20:48:04 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:13:01 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.4,},NodeAddress{Type:Hostname,Address:k8s-master-35271922-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:8843d4f5cd0544d08ae501580db6a3a7,SystemUUID:41DCC090-746D-9E44-8888-8A77F0E0C51C,BootID:42d9eb02-ab49-4c2a-8791-7c5b2ac8f2bc,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:45e5b123e6188348715630acd2f7c1469c5b5181078ce90dc6f5399be4c17be1 k8sprow.azurecr.io/azure-cloud-controller-manager:1193984044413489154-9318bf1d],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:13:35.506: INFO: Logging kubelet events for node k8s-master-35271922-vmss000000 Nov 11 21:13:35.563: INFO: Logging pods the kubelet thinks is on node k8s-master-35271922-vmss000000 Nov 11 21:13:35.651: INFO: kube-proxy-8hzbt started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:35.651: INFO: Container kube-proxy ready: true, restart count 0 Nov 11 21:13:35.651: INFO: cloud-controller-manager-k8s-master-35271922-vmss000000 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:35.651: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 11 21:13:35.651: INFO: kube-addon-manager-k8s-master-35271922-vmss000000 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:35.651: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 11 21:13:35.651: INFO: kube-apiserver-k8s-master-35271922-vmss000000 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:35.651: INFO: Container kube-apiserver ready: true, restart count 0 Nov 11 21:13:35.651: INFO: kube-controller-manager-k8s-master-35271922-vmss000000 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:35.651: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 11 21:13:35.652: INFO: kube-scheduler-k8s-master-35271922-vmss000000 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:35.652: INFO: Container kube-scheduler ready: true, restart count 0 Nov 11 21:13:35.652: INFO: azure-ip-masq-agent-k7l5b started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:35.652: INFO: Container azure-ip-masq-agent ready: true, restart count 0 W1111 21:13:35.705541 93386 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:13:35.821: INFO: Latency metrics for node k8s-master-35271922-vmss000000 Nov 11 21:13:35.821: INFO: Logging node info for node k8s-master-35271922-vmss000001 Nov 11 21:13:35.874: INFO: Node Info: &Node{ObjectMeta:{k8s-master-35271922-vmss000001 /api/v1/nodes/k8s-master-35271922-vmss000001 0ff06487-2e4c-4c07-a6f2-d77d1a47bbcd 41814 0 2019-11-11 20:48:39 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-35271922-vmss000001 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.6.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-35271922-vmss/virtualMachines/1,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.6.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284363264 0} {<nil>} 7113636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6497931264 0} {<nil>} 6345636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:49:01 +0000 UTC,LastTransitionTime:2019-11-11 20:49:01 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:13:04 +0000 UTC,LastTransitionTime:2019-11-11 20:48:32 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:13:04 +0000 UTC,LastTransitionTime:2019-11-11 20:48:32 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:13:04 +0000 UTC,LastTransitionTime:2019-11-11 20:48:32 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:13:04 +0000 UTC,LastTransitionTime:2019-11-11 20:48:39 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.5,},NodeAddress{Type:Hostname,Address:k8s-master-35271922-vmss000001,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:24b17940193c48e38f59d10d619fb718,SystemUUID:25FDA250-5862-9148-98C8-17FFDD874AEC,BootID:cdd7987a-6ece-464d-a177-21f53ae0b68b,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:45e5b123e6188348715630acd2f7c1469c5b5181078ce90dc6f5399be4c17be1 k8sprow.azurecr.io/azure-cloud-controller-manager:1193984044413489154-9318bf1d],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:13:35.875: INFO: Logging kubelet events for node k8s-master-35271922-vmss000001 Nov 11 21:13:35.931: INFO: Logging pods the kubelet thinks is on node k8s-master-35271922-vmss000001 Nov 11 21:13:36.010: INFO: azure-ip-masq-agent-bm2t5 started at 2019-11-11 20:48:40 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:36.010: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:13:36.010: INFO: kube-proxy-5fcm6 started at 2019-11-11 20:48:40 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:36.010: INFO: Container kube-proxy ready: true, restart count 0 Nov 11 21:13:36.010: INFO: cloud-controller-manager-k8s-master-35271922-vmss000001 started at 2019-11-11 20:48:26 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:36.010: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 11 21:13:36.010: INFO: kube-addon-manager-k8s-master-35271922-vmss000001 started at 2019-11-11 20:48:26 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:36.010: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 11 21:13:36.010: INFO: kube-apiserver-k8s-master-35271922-vmss000001 started at 2019-11-11 20:48:26 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:36.010: INFO: Container kube-apiserver ready: true, restart count 0 Nov 11 21:13:36.010: INFO: kube-controller-manager-k8s-master-35271922-vmss000001 started at 2019-11-11 20:48:26 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:36.010: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 11 21:13:36.010: INFO: kube-scheduler-k8s-master-35271922-vmss000001 started at 2019-11-11 20:48:26 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:36.010: INFO: Container kube-scheduler ready: true, restart count 0 W1111 21:13:36.063404 93386 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:13:36.181: INFO: Latency metrics for node k8s-master-35271922-vmss000001 Nov 11 21:13:36.181: INFO: Logging node info for node k8s-master-35271922-vmss000002 Nov 11 21:13:36.233: INFO: Node Info: &Node{ObjectMeta:{k8s-master-35271922-vmss000002 /api/v1/nodes/k8s-master-35271922-vmss000002 7fedbabf-dce6-48d1-994b-628369e7bcb5 41799 0 2019-11-11 20:48:08 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-35271922-vmss000002 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.4.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-35271922-vmss/virtualMachines/2,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284363264 0} {<nil>} 7113636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6497931264 0} {<nil>} 6345636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:13:03 +0000 UTC,LastTransitionTime:2019-11-11 20:48:03 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:13:03 +0000 UTC,LastTransitionTime:2019-11-11 20:48:03 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:13:03 +0000 UTC,LastTransitionTime:2019-11-11 20:48:03 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:13:03 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.6,},NodeAddress{Type:Hostname,Address:k8s-master-35271922-vmss000002,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:5ecd4c723b8845b18d85d341ba062204,SystemUUID:9E772AEA-D3D7-084D-98ED-1C1A511168B4,BootID:a73a4655-81c3-42e2-9d20-ef7e3cff3664,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:45e5b123e6188348715630acd2f7c1469c5b5181078ce90dc6f5399be4c17be1 k8sprow.azurecr.io/azure-cloud-controller-manager:1193984044413489154-9318bf1d],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:13:36.234: INFO: Logging kubelet events for node k8s-master-35271922-vmss000002 Nov 11 21:13:36.289: INFO: Logging pods the kubelet thinks is on node k8s-master-35271922-vmss000002 Nov 11 21:13:36.376: INFO: cloud-controller-manager-k8s-master-35271922-vmss000002 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:36.376: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 11 21:13:36.376: INFO: kube-addon-manager-k8s-master-35271922-vmss000002 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:36.376: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 11 21:13:36.376: INFO: kube-apiserver-k8s-master-35271922-vmss000002 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:36.376: INFO: Container kube-apiserver ready: true, restart count 0 Nov 11 21:13:36.376: INFO: kube-controller-manager-k8s-master-35271922-vmss000002 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:36.376: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 11 21:13:36.376: INFO: kube-scheduler-k8s-master-35271922-vmss000002 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:36.376: INFO: Container kube-scheduler ready: true, restart count 0 Nov 11 21:13:36.376: INFO: azure-ip-masq-agent-cmbqm started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:36.376: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:13:36.376: INFO: kube-proxy-t2sp7 started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:36.376: INFO: Container kube-proxy ready: true, restart count 0 W1111 21:13:36.429951 93386 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:13:36.553: INFO: Latency metrics for node k8s-master-35271922-vmss000002 Nov 11 21:13:36.553: INFO: Logging node info for node k8s-master-35271922-vmss000003 Nov 11 21:13:36.604: INFO: Node Info: &Node{ObjectMeta:{k8s-master-35271922-vmss000003 /api/v1/nodes/k8s-master-35271922-vmss000003 d4d04c51-fc7b-4cdc-a1e3-69990a198568 41819 0 2019-11-11 20:48:10 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-35271922-vmss000003 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.5.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-35271922-vmss/virtualMachines/3,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.5.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284363264 0} {<nil>} 7113636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6497931264 0} {<nil>} 6345636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:13:04 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:13:04 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:13:04 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:13:04 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.7,},NodeAddress{Type:Hostname,Address:k8s-master-35271922-vmss000003,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:53117d11ac104f35951d19e4a4448f8f,SystemUUID:048B9870-F8AE-BC47-80D8-26703FED9760,BootID:fc64edd8-d9c5-4d6f-b9e7-c902146bc724,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:45e5b123e6188348715630acd2f7c1469c5b5181078ce90dc6f5399be4c17be1 k8sprow.azurecr.io/azure-cloud-controller-manager:1193984044413489154-9318bf1d],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:13:36.605: INFO: Logging kubelet events for node k8s-master-35271922-vmss000003 Nov 11 21:13:36.659: INFO: Logging pods the kubelet thinks is on node k8s-master-35271922-vmss000003 Nov 11 21:13:36.736: INFO: cloud-controller-manager-k8s-master-35271922-vmss000003 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:36.736: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 11 21:13:36.737: INFO: kube-addon-manager-k8s-master-35271922-vmss000003 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:36.737: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 11 21:13:36.737: INFO: kube-apiserver-k8s-master-35271922-vmss000003 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:36.737: INFO: Container kube-apiserver ready: true, restart count 0 Nov 11 21:13:36.737: INFO: kube-controller-manager-k8s-master-35271922-vmss000003 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:36.737: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 11 21:13:36.737: INFO: kube-scheduler-k8s-master-35271922-vmss000003 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:36.737: INFO: Container kube-scheduler ready: true, restart count 0 Nov 11 21:13:36.737: INFO: azure-ip-masq-agent-rs668 started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:36.737: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:13:36.737: INFO: kube-proxy-tqjkf started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:36.737: INFO: Container kube-proxy ready: true, restart count 0 W1111 21:13:36.796641 93386 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:13:36.913: INFO: Latency metrics for node k8s-master-35271922-vmss000003 Nov 11 21:13:36.913: INFO: Logging node info for node k8s-master-35271922-vmss000004 Nov 11 21:13:36.965: INFO: Node Info: &Node{ObjectMeta:{k8s-master-35271922-vmss000004 /api/v1/nodes/k8s-master-35271922-vmss000004 edf29ad2-2fc8-4411-ac05-e1bf8781e340 41845 0 2019-11-11 20:48:11 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-35271922-vmss000004 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-35271922-vmss/virtualMachines/4,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284363264 0} {<nil>} 7113636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6497931264 0} {<nil>} 6345636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:13:05 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:13:05 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:13:05 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:13:05 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.8,},NodeAddress{Type:Hostname,Address:k8s-master-35271922-vmss000004,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:3ecc95ca67e14e2e87659efc6707e3b4,SystemUUID:0649BB2F-7C03-5743-A309-9BABFF3DD11A,BootID:1256b8d0-bf8e-4490-a45d-beaf41e45a09,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:45e5b123e6188348715630acd2f7c1469c5b5181078ce90dc6f5399be4c17be1 k8sprow.azurecr.io/azure-cloud-controller-manager:1193984044413489154-9318bf1d],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:13:36.966: INFO: Logging kubelet events for node k8s-master-35271922-vmss000004 Nov 11 21:13:37.022: INFO: Logging pods the kubelet thinks is on node k8s-master-35271922-vmss000004 Nov 11 21:13:37.101: INFO: kube-controller-manager-k8s-master-35271922-vmss000004 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:37.101: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 11 21:13:37.101: INFO: kube-scheduler-k8s-master-35271922-vmss000004 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:37.101: INFO: Container kube-scheduler ready: true, restart count 0 Nov 11 21:13:37.101: INFO: azure-ip-masq-agent-trzsm started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:37.101: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:13:37.101: INFO: kube-proxy-zrmqq started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:37.101: INFO: Container kube-proxy ready: true, restart count 0 Nov 11 21:13:37.101: INFO: cloud-controller-manager-k8s-master-35271922-vmss000004 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:37.101: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 11 21:13:37.101: INFO: kube-addon-manager-k8s-master-35271922-vmss000004 started at 2019-11-11 20:48:19 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:37.101: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 11 21:13:37.101: INFO: kube-apiserver-k8s-master-35271922-vmss000004 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:37.101: INFO: Container kube-apiserver ready: true, restart count 0 W1111 21:13:37.154349 93386 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:13:37.269: INFO: Latency metrics for node k8s-master-35271922-vmss000004 Nov 11 21:13:37.269: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "volume-expand-5804" for this suite. WARNING: pod log: csi-snapshotter-0/csi-snapshotter: pods "csi-snapshotter-0" not found Nov 11 21:15:49.488: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered Nov 11 21:15:51.297: INFO: namespace volume-expand-5804 deletion completed in 2m13.974687053s
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[sig\-storage\]\sCSI\sVolumes\s\[Driver\:\scsi\-hostpath\]\s\[Testpattern\:\sDynamic\sPV\s\(block\svolmode\)\]\svolumes\sshould\sstore\sdata$'
test/e2e/storage/testsuites/volumes.go:146 Nov 11 21:23:10.309: Unexpected error: <*errors.errorString | 0xc0008a3ba0>: { s: "PersistentVolumeClaims [csi-hostpathrcfkh] not all in phase Bound within 5m0s", } PersistentVolumeClaims [csi-hostpathrcfkh] not all in phase Bound within 5m0s occurred test/e2e/storage/testsuites/base.go:366from junit_09.xml
[BeforeEach] [Testpattern: Dynamic PV (block volmode)] volumes test/e2e/storage/testsuites/base.go:93 [BeforeEach] [Testpattern: Dynamic PV (block volmode)] volumes test/e2e/framework/framework.go:151 �[1mSTEP�[0m: Creating a kubernetes client Nov 11 21:18:07.369: INFO: >>> kubeConfig: /workspace/aks147280421/kubeconfig/kubeconfig.westus2.json �[1mSTEP�[0m: Building a namespace api object, basename volume �[1mSTEP�[0m: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in volume-8245 �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace [It] should store data test/e2e/storage/testsuites/volumes.go:146 �[1mSTEP�[0m: deploying csi-hostpath driver Nov 11 21:18:07.936: INFO: creating *v1.ServiceAccount: volume-8245/csi-attacher Nov 11 21:18:07.994: INFO: creating *v1.ClusterRole: external-attacher-runner-volume-8245 Nov 11 21:18:07.994: INFO: Define cluster role external-attacher-runner-volume-8245 Nov 11 21:18:08.050: INFO: creating *v1.ClusterRoleBinding: csi-attacher-role-volume-8245 Nov 11 21:18:08.108: INFO: creating *v1.Role: volume-8245/external-attacher-cfg-volume-8245 Nov 11 21:18:08.164: INFO: creating *v1.RoleBinding: volume-8245/csi-attacher-role-cfg Nov 11 21:18:08.220: INFO: creating *v1.ServiceAccount: volume-8245/csi-provisioner Nov 11 21:18:08.279: INFO: creating *v1.ClusterRole: external-provisioner-runner-volume-8245 Nov 11 21:18:08.280: INFO: Define cluster role external-provisioner-runner-volume-8245 Nov 11 21:18:08.336: INFO: creating *v1.ClusterRoleBinding: csi-provisioner-role-volume-8245 Nov 11 21:18:08.398: INFO: creating *v1.Role: volume-8245/external-provisioner-cfg-volume-8245 Nov 11 21:18:08.465: INFO: creating *v1.RoleBinding: volume-8245/csi-provisioner-role-cfg Nov 11 21:18:08.528: INFO: creating *v1.ServiceAccount: volume-8245/csi-snapshotter Nov 11 21:18:08.584: INFO: creating *v1.ClusterRole: external-snapshotter-runner-volume-8245 Nov 11 21:18:08.584: INFO: Define cluster role external-snapshotter-runner-volume-8245 Nov 11 21:18:08.641: INFO: creating *v1.ClusterRoleBinding: csi-snapshotter-role-volume-8245 Nov 11 21:18:08.697: INFO: creating *v1.Role: volume-8245/external-snapshotter-leaderelection-volume-8245 Nov 11 21:18:08.753: INFO: creating *v1.RoleBinding: volume-8245/external-snapshotter-leaderelection Nov 11 21:18:08.809: INFO: creating *v1.ServiceAccount: volume-8245/csi-resizer Nov 11 21:18:08.865: INFO: creating *v1.ClusterRole: external-resizer-runner-volume-8245 Nov 11 21:18:08.865: INFO: Define cluster role external-resizer-runner-volume-8245 Nov 11 21:18:08.924: INFO: creating *v1.ClusterRoleBinding: csi-resizer-role-volume-8245 Nov 11 21:18:09.023: INFO: creating *v1.Role: volume-8245/external-resizer-cfg-volume-8245 Nov 11 21:18:09.081: INFO: creating *v1.RoleBinding: volume-8245/csi-resizer-role-cfg Nov 11 21:18:09.138: INFO: creating *v1.Service: volume-8245/csi-hostpath-attacher Nov 11 21:18:09.220: INFO: creating *v1.StatefulSet: volume-8245/csi-hostpath-attacher Nov 11 21:18:09.285: INFO: creating *v1beta1.CSIDriver: csi-hostpath-volume-8245 Nov 11 21:18:09.341: INFO: creating *v1.Service: volume-8245/csi-hostpathplugin Nov 11 21:18:09.464: INFO: creating *v1.StatefulSet: volume-8245/csi-hostpathplugin Nov 11 21:18:09.536: INFO: creating *v1.Service: volume-8245/csi-hostpath-provisioner Nov 11 21:18:09.611: INFO: creating *v1.StatefulSet: volume-8245/csi-hostpath-provisioner Nov 11 21:18:09.676: INFO: creating *v1.Service: volume-8245/csi-hostpath-resizer Nov 11 21:18:09.748: INFO: creating *v1.StatefulSet: volume-8245/csi-hostpath-resizer Nov 11 21:18:09.807: INFO: creating *v1.Service: volume-8245/csi-snapshotter Nov 11 21:18:09.877: INFO: creating *v1.StatefulSet: volume-8245/csi-snapshotter Nov 11 21:18:09.938: INFO: creating *v1.ClusterRoleBinding: psp-csi-hostpath-role-volume-8245 Nov 11 21:18:10.003: INFO: Test running for native CSI Driver, not checking metrics Nov 11 21:18:10.003: INFO: Creating resource for dynamic PV �[1mSTEP�[0m: creating a StorageClass volume-8245-csi-hostpath-volume-8245-sczn5fw �[1mSTEP�[0m: creating a claim Nov 11 21:18:10.160: INFO: Waiting up to 5m0s for PersistentVolumeClaims [csi-hostpathrcfkh] to have phase Bound Nov 11 21:18:10.215: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:18:12.268: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:18:14.323: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:18:16.381: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:18:18.436: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:18:20.489: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:18:22.551: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:18:24.605: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:18:26.659: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:18:28.712: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:18:30.765: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:18:32.824: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:18:34.879: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:18:36.932: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:18:38.985: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:18:41.046: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:18:43.116: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:18:45.170: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:18:47.224: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:18:49.277: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:18:51.331: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:18:53.385: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:18:55.444: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:18:57.497: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:18:59.555: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:19:01.610: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:19:03.663: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:19:05.720: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:19:07.773: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:19:09.831: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:19:11.888: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:19:13.941: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:19:15.994: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:19:18.047: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:19:20.112: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:19:22.165: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:19:24.219: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:19:26.273: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:19:28.327: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:19:30.379: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:19:32.432: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:19:34.487: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:19:36.540: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:19:38.594: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:19:40.647: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:19:42.701: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:19:44.755: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:19:46.808: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:19:48.868: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:19:50.925: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:19:53.007: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:19:55.061: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:19:57.114: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:19:59.167: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:20:01.221: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:20:03.274: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:20:05.327: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:20:07.380: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:20:09.445: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:20:11.497: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:20:13.556: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:20:15.609: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:20:17.663: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:20:19.731: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:20:21.796: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:20:23.849: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:20:25.902: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:20:27.956: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:20:30.015: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:20:32.068: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:20:34.124: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:20:36.205: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:20:38.258: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:20:40.311: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:20:42.364: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:20:44.418: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:20:46.472: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:20:48.526: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:20:50.579: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:20:52.632: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:20:54.685: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:20:56.741: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:20:58.795: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:21:00.848: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:21:02.902: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:21:04.955: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:21:07.009: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:21:09.062: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:21:11.116: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:21:13.170: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:21:15.222: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:21:17.275: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:21:19.332: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:21:21.385: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:21:23.438: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:21:25.492: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:21:27.552: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:21:29.609: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:21:31.673: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:21:33.726: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:21:35.779: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:21:37.832: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:21:39.885: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:21:41.944: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:21:43.997: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:21:46.052: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:21:48.106: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:21:50.159: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:21:52.213: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:21:54.267: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:21:56.324: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:21:58.379: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:22:00.433: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:22:02.486: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:22:04.538: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:22:06.593: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:22:08.646: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:22:10.698: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:22:12.760: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:22:14.818: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:22:16.872: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:22:18.928: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:22:21.016: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:22:23.074: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:22:25.128: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:22:27.181: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:22:29.234: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:22:31.288: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:22:33.341: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:22:35.395: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:22:37.450: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:22:39.504: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:22:41.564: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:22:43.620: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:22:45.673: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:22:47.729: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:22:49.787: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:22:51.840: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:22:53.893: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:22:55.950: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:22:58.003: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:23:00.057: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:23:02.149: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:23:04.203: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:23:06.256: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:23:08.309: INFO: PersistentVolumeClaim csi-hostpathrcfkh found but phase is Pending instead of Bound. Nov 11 21:23:10.309: FAIL: Unexpected error: <*errors.errorString | 0xc0008a3ba0>: { s: "PersistentVolumeClaims [csi-hostpathrcfkh] not all in phase Bound within 5m0s", } PersistentVolumeClaims [csi-hostpathrcfkh] not all in phase Bound within 5m0s occurred [AfterEach] [Testpattern: Dynamic PV (block volmode)] volumes test/e2e/framework/framework.go:152 �[1mSTEP�[0m: Collecting events from namespace "volume-8245". �[1mSTEP�[0m: Found 51 events. Nov 11 21:23:10.364: INFO: At 2019-11-11 21:18:09 +0000 UTC - event for csi-hostpath-attacher: {statefulset-controller } SuccessfulCreate: create Pod csi-hostpath-attacher-0 in StatefulSet csi-hostpath-attacher successful Nov 11 21:23:10.364: INFO: At 2019-11-11 21:18:09 +0000 UTC - event for csi-hostpath-provisioner: {statefulset-controller } SuccessfulCreate: create Pod csi-hostpath-provisioner-0 in StatefulSet csi-hostpath-provisioner successful Nov 11 21:23:10.364: INFO: At 2019-11-11 21:18:09 +0000 UTC - event for csi-hostpath-resizer: {statefulset-controller } SuccessfulCreate: create Pod csi-hostpath-resizer-0 in StatefulSet csi-hostpath-resizer successful Nov 11 21:23:10.364: INFO: At 2019-11-11 21:18:09 +0000 UTC - event for csi-hostpathplugin: {statefulset-controller } SuccessfulCreate: create Pod csi-hostpathplugin-0 in StatefulSet csi-hostpathplugin successful Nov 11 21:23:10.364: INFO: At 2019-11-11 21:18:09 +0000 UTC - event for csi-snapshotter: {statefulset-controller } SuccessfulCreate: create Pod csi-snapshotter-0 in StatefulSet csi-snapshotter successful Nov 11 21:23:10.364: INFO: At 2019-11-11 21:18:10 +0000 UTC - event for csi-hostpathrcfkh: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "csi-hostpath-volume-8245" or manually created by system administrator Nov 11 21:23:10.364: INFO: At 2019-11-11 21:18:12 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:23:10.364: INFO: At 2019-11-11 21:18:22 +0000 UTC - event for csi-hostpathplugin: {statefulset-controller } SuccessfulDelete: delete Pod csi-hostpathplugin-0 in StatefulSet csi-hostpathplugin successful Nov 11 21:23:10.364: INFO: At 2019-11-11 21:18:22 +0000 UTC - event for csi-hostpathplugin: {statefulset-controller } RecreatingFailedPod: StatefulSet volume-8245/csi-hostpathplugin is recreating failed Pod csi-hostpathplugin-0 Nov 11 21:23:10.364: INFO: At 2019-11-11 21:18:24 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:23:10.365: INFO: At 2019-11-11 21:18:26 +0000 UTC - event for csi-hostpath-attacher-0: {kubelet k8s-agentpool-35271922-vmss000000} Pulling: Pulling image "quay.io/k8scsi/csi-attacher:v1.2.0" Nov 11 21:23:10.365: INFO: At 2019-11-11 21:18:27 +0000 UTC - event for csi-snapshotter-0: {kubelet k8s-agentpool-35271922-vmss000000} Pulling: Pulling image "quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1" Nov 11 21:23:10.365: INFO: At 2019-11-11 21:18:30 +0000 UTC - event for csi-hostpath-provisioner-0: {kubelet k8s-agentpool-35271922-vmss000000} Pulling: Pulling image "quay.io/k8scsi/csi-provisioner:v1.4.0-rc1" Nov 11 21:23:10.365: INFO: At 2019-11-11 21:18:31 +0000 UTC - event for csi-hostpath-attacher-0: {kubelet k8s-agentpool-35271922-vmss000000} Pulled: Successfully pulled image "quay.io/k8scsi/csi-attacher:v1.2.0" Nov 11 21:23:10.365: INFO: At 2019-11-11 21:18:31 +0000 UTC - event for csi-hostpath-resizer-0: {kubelet k8s-agentpool-35271922-vmss000000} Pulling: Pulling image "quay.io/k8scsi/csi-resizer:v0.2.0" Nov 11 21:23:10.365: INFO: At 2019-11-11 21:18:32 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:23:10.365: INFO: At 2019-11-11 21:18:35 +0000 UTC - event for csi-snapshotter-0: {kubelet k8s-agentpool-35271922-vmss000000} Pulled: Successfully pulled image "quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1" Nov 11 21:23:10.365: INFO: At 2019-11-11 21:18:36 +0000 UTC - event for csi-hostpath-attacher-0: {kubelet k8s-agentpool-35271922-vmss000000} Created: Created container csi-attacher Nov 11 21:23:10.365: INFO: At 2019-11-11 21:18:39 +0000 UTC - event for csi-hostpath-provisioner-0: {kubelet k8s-agentpool-35271922-vmss000000} Pulled: Successfully pulled image "quay.io/k8scsi/csi-provisioner:v1.4.0-rc1" Nov 11 21:23:10.365: INFO: At 2019-11-11 21:18:39 +0000 UTC - event for csi-snapshotter-0: {kubelet k8s-agentpool-35271922-vmss000000} Created: Created container csi-snapshotter Nov 11 21:23:10.365: INFO: At 2019-11-11 21:18:40 +0000 UTC - event for csi-hostpath-attacher-0: {kubelet k8s-agentpool-35271922-vmss000000} Started: Started container csi-attacher Nov 11 21:23:10.365: INFO: At 2019-11-11 21:18:40 +0000 UTC - event for csi-hostpathplugin: {statefulset-controller } FailedCreate: create Pod csi-hostpathplugin-0 in StatefulSet csi-hostpathplugin failed error: The POST operation against Pod could not be completed at this time, please try again. Nov 11 21:23:10.365: INFO: At 2019-11-11 21:18:40 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:23:10.365: INFO: At 2019-11-11 21:18:42 +0000 UTC - event for csi-snapshotter-0: {kubelet k8s-agentpool-35271922-vmss000000} Started: Started container csi-snapshotter Nov 11 21:23:10.365: INFO: At 2019-11-11 21:18:43 +0000 UTC - event for csi-hostpath-provisioner-0: {kubelet k8s-agentpool-35271922-vmss000000} Created: Created container csi-provisioner Nov 11 21:23:10.365: INFO: At 2019-11-11 21:18:43 +0000 UTC - event for csi-hostpath-resizer-0: {kubelet k8s-agentpool-35271922-vmss000000} Pulled: Successfully pulled image "quay.io/k8scsi/csi-resizer:v0.2.0" Nov 11 21:23:10.365: INFO: At 2019-11-11 21:18:47 +0000 UTC - event for csi-hostpath-provisioner-0: {kubelet k8s-agentpool-35271922-vmss000000} Started: Started container csi-provisioner Nov 11 21:23:10.365: INFO: At 2019-11-11 21:18:49 +0000 UTC - event for csi-hostpath-resizer-0: {kubelet k8s-agentpool-35271922-vmss000000} Created: Created container csi-resizer Nov 11 21:23:10.365: INFO: At 2019-11-11 21:18:51 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:23:10.365: INFO: At 2019-11-11 21:18:52 +0000 UTC - event for csi-hostpath-resizer-0: {kubelet k8s-agentpool-35271922-vmss000000} Started: Started container csi-resizer Nov 11 21:23:10.365: INFO: At 2019-11-11 21:18:55 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:23:10.365: INFO: At 2019-11-11 21:19:00 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:23:10.365: INFO: At 2019-11-11 21:19:03 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:23:10.365: INFO: At 2019-11-11 21:19:13 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:23:10.365: INFO: At 2019-11-11 21:19:19 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:23:10.365: INFO: At 2019-11-11 21:19:35 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:23:10.365: INFO: At 2019-11-11 21:19:52 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:23:10.365: INFO: At 2019-11-11 21:20:07 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:23:10.365: INFO: At 2019-11-11 21:20:13 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:23:10.365: INFO: At 2019-11-11 21:20:25 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:23:10.365: INFO: At 2019-11-11 21:20:43 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:23:10.365: INFO: At 2019-11-11 21:21:01 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:23:10.365: INFO: At 2019-11-11 21:21:33 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:23:10.365: INFO: At 2019-11-11 21:21:48 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:23:10.365: INFO: At 2019-11-11 21:21:59 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:23:10.365: INFO: At 2019-11-11 21:22:10 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:23:10.365: INFO: At 2019-11-11 21:22:22 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:23:10.365: INFO: At 2019-11-11 21:22:32 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:23:10.365: INFO: At 2019-11-11 21:22:41 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:23:10.365: INFO: At 2019-11-11 21:22:53 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:23:10.365: INFO: At 2019-11-11 21:23:02 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:23:10.437: INFO: POD NODE PHASE GRACE CONDITIONS Nov 11 21:23:10.437: INFO: csi-hostpath-attacher-0 k8s-agentpool-35271922-vmss000000 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:18:12 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:18:44 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:18:44 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:18:12 +0000 UTC }] Nov 11 21:23:10.437: INFO: csi-hostpath-provisioner-0 k8s-agentpool-35271922-vmss000000 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:18:13 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:18:52 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:18:52 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:18:13 +0000 UTC }] Nov 11 21:23:10.437: INFO: csi-hostpath-resizer-0 k8s-agentpool-35271922-vmss000000 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:18:13 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:18:55 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:18:55 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:18:13 +0000 UTC }] Nov 11 21:23:10.437: INFO: csi-hostpathplugin-0 k8s-agentpool-35271922-vmss000000 Pending [] Nov 11 21:23:10.437: INFO: csi-snapshotter-0 k8s-agentpool-35271922-vmss000000 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:18:13 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:18:47 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:18:47 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:18:13 +0000 UTC }] Nov 11 21:23:10.437: INFO: Nov 11 21:23:10.546: INFO: Logging node info for node k8s-agentpool-35271922-vmss000000 Nov 11 21:23:10.598: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool-35271922-vmss000000 /api/v1/nodes/k8s-agentpool-35271922-vmss000000 04da751e-f23a-470e-b29d-e591113d7271 53507 0 2019-11-11 20:48:07 +0000 UTC <nil> <nil> map[agentpool:agentpool beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool-35271922-vmss000000 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: storageprofile:managed storagetier:Premium_LRS] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-2030":"k8s-agentpool-35271922-vmss000000","csi-hostpath-ephemeral-7436":"k8s-agentpool-35271922-vmss000000","csi-hostpath-provisioning-239":"k8s-agentpool-35271922-vmss000000","csi-hostpath-provisioning-3597":"k8s-agentpool-35271922-vmss000000","csi-hostpath-provisioning-9334":"k8s-agentpool-35271922-vmss000000","csi-hostpath-volume-5092":"k8s-agentpool-35271922-vmss000000","csi-hostpath-volume-expand-1449":"k8s-agentpool-35271922-vmss000000","csi-hostpath-volume-expand-5724":"k8s-agentpool-35271922-vmss000000","csi-hostpath-volume-expand-5832":"k8s-agentpool-35271922-vmss000000","csi-mock-csi-mock-volumes-1804":"csi-mock-csi-mock-volumes-1804","csi-mock-csi-mock-volumes-1829":"csi-mock-csi-mock-volumes-1829","csi-mock-csi-mock-volumes-2481":"csi-mock-csi-mock-volumes-2481","csi-mock-csi-mock-volumes-3755":"csi-mock-csi-mock-volumes-3755","csi-mock-csi-mock-volumes-763":"csi-mock-csi-mock-volumes-763","csi-mock-csi-mock-volumes-9721":"csi-mock-csi-mock-volumes-9721"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-35271922-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16797044736 0} {<nil>} 16403364Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16010612736 0} {<nil>} 15635364Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:22:55 +0000 UTC,LastTransitionTime:2019-11-11 20:47:58 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:22:55 +0000 UTC,LastTransitionTime:2019-11-11 20:47:58 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:22:55 +0000 UTC,LastTransitionTime:2019-11-11 20:47:58 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:22:55 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.248.0.4,},NodeAddress{Type:Hostname,Address:k8s-agentpool-35271922-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:84c06f3ee41a47aaa558c81f34711895,SystemUUID:9BB24974-637B-EE4C-A221-0F74BB743424,BootID:59f7eb0c-c09c-4eae-a14b-0710f9e14004,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[quay.io/kubernetes_incubator/nfs-provisioner@sha256:df762117e3c891f2d2ddff46ecb0776ba1f9f3c44cfd7739b0683bcd7a7954a8 quay.io/kubernetes_incubator/nfs-provisioner:v2.2.2],SizeBytes:391772778,},ContainerImage{Names:[gluster/glusterdynamic-provisioner@sha256:90067cb05a7d217651e84576935934fd9dcff8e6bcdcbaa416bbf36fcd09dbd1 gluster/glusterdynamic-provisioner:v1.0],SizeBytes:373281573,},ContainerImage{Names:[gcr.io/google-samples/gb-frontend@sha256:35cb427341429fac3df10ff74600ea73e8ec0754d78f9ce89e0b4f3d70d53ba6 gcr.io/google-samples/gb-frontend:v6],SizeBytes:373099368,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/volume/nfs@sha256:c2ad734346f608a5f7d69cfded93c4e8094069320657bd372d12ba21dea3ea71 gcr.io/kubernetes-e2e-test-images/volume/nfs:1.0],SizeBytes:225358913,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/jessie-dnsutils@sha256:ad583e33cb284f7ef046673809b146ec4053cda19b54a85d2b180a86169715eb gcr.io/kubernetes-e2e-test-images/jessie-dnsutils:1.0],SizeBytes:195659796,},ContainerImage{Names:[httpd@sha256:addd70e4ee83f3bc9a4c1c7c41e37927ba47faf639312fc936df3afad7926f5a httpd:2.4.39-alpine],SizeBytes:126894770,},ContainerImage{Names:[httpd@sha256:eb8ccf084cf3e80eece1add239effefd171eb39adbc154d33c14260d905d4060 httpd:2.4.38-alpine],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1],SizeBytes:121711221,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727 gcr.io/kubernetes-e2e-test-images/agnhost:2.6],SizeBytes:57345321,},ContainerImage{Names:[quay.io/k8scsi/csi-provisioner@sha256:0efcb424f1dde9b9fb11a1a14f2e48ab47e1c3f08bc3a929990dcfcb1f7ab34f quay.io/k8scsi/csi-provisioner:v1.4.0-rc1],SizeBytes:54431016,},ContainerImage{Names:[quay.io/k8scsi/csi-snapshotter@sha256:e3d3e742e32d00488fdb401045b9b1d033d7ca0ab6e760f77b24750fc95e5f70 quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1],SizeBytes:51703561,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0],SizeBytes:46226754,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:0aba670b4d9d6b2e720bbf575d733156c676b693ca26501235444490300db838 quay.io/k8scsi/csi-attacher:v1.1.0],SizeBytes:42839085,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:7d46fb6eb8b890dc546029d1565d502b4a1d974d33625c6ee2bc7991b77fc1a1 quay.io/k8scsi/csi-resizer:v0.2.0],SizeBytes:42817100,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:f315c9042e56def3c05c6b04fe79ec9da6d39ddc557ca365a76cf35964ea08b6 quay.io/k8scsi/csi-resizer:v0.1.0],SizeBytes:42623056,},ContainerImage{Names:[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0],SizeBytes:42488424,},ContainerImage{Names:[redis@sha256:50899ea1ceed33fa03232f3ac57578a424faa1742c1ac9c7a7bdb95cdf19b858 redis:5.0.5-alpine],SizeBytes:29331594,},ContainerImage{Names:[quay.io/k8scsi/hostpathplugin@sha256:b4826e492fc1762fceaf9726f41575ca0a4567864d3d235da874818de18039de quay.io/k8scsi/hostpathplugin:v1.2.0-rc5],SizeBytes:28761497,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7],SizeBytes:22909487,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/echoserver@sha256:e9ba514b896cdf559eef8788b66c2c3ee55f3572df617647b4b0d8b6bf81cf19 gcr.io/kubernetes-e2e-test-images/echoserver:2.2],SizeBytes:21692741,},ContainerImage{Names:[quay.io/k8scsi/mock-driver@sha256:e0eed916b7d970bad2b7d9875f9ad16932f987f0f3d91ec5d86da68b0b5cc9d1 quay.io/k8scsi/mock-driver:v2.1.0],SizeBytes:16226335,},ContainerImage{Names:[nginx@sha256:485b610fefec7ff6c463ced9623314a04ed67e3945b9c08d7e53a47f6d108dc7 nginx:1.14-alpine],SizeBytes:16032814,},ContainerImage{Names:[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0],SizeBytes:15815995,},ContainerImage{Names:[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0],SizeBytes:14967303,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/dnsutils@sha256:2abeee84efb79c14d731966e034af33bf324d3b26ca28497555511ff094b3ddd gcr.io/kubernetes-e2e-test-images/dnsutils:1.1],SizeBytes:9349974,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nautilus@sha256:33a732d4c42a266912a5091598a0f07653c9134db4b8d571690d8afd509e0bfc gcr.io/kubernetes-e2e-test-images/nautilus:1.0],SizeBytes:4753501,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/kitten@sha256:bcbc4875c982ab39aa7c4f6acf4a287f604e996d9f34a3fbda8c3d1a7457d1f6 gcr.io/kubernetes-e2e-test-images/kitten:1.0],SizeBytes:4747037,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/test-webserver@sha256:7f93d6e32798ff28bc6289254d0c2867fe2c849c8e46edc50f8624734309812e gcr.io/kubernetes-e2e-test-images/test-webserver:1.0],SizeBytes:4732240,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 gcr.io/kubernetes-e2e-test-images/mounttest:1.0],SizeBytes:1563521,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29],SizeBytes:1154361,},ContainerImage{Names:[busybox@sha256:bbc3a03235220b170ba48a157dd097dd1379299370e1ed99ce976df0355d24f0 busybox:1.27],SizeBytes:1129289,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea k8s.gcr.io/pause-amd64:3.1 k8s.gcr.io/pause:3.1],SizeBytes:742472,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-provisioning-239^65c5c89d-04c9-11ea-a9c8-000d3af7321e],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-provisioning-239^65c5c89d-04c9-11ea-a9c8-000d3af7321e,DevicePath:,},},Config:nil,},} Nov 11 21:23:10.599: INFO: Logging kubelet events for node k8s-agentpool-35271922-vmss000000 Nov 11 21:23:10.661: INFO: Logging pods the kubelet thinks is on node k8s-agentpool-35271922-vmss000000 Nov 11 21:23:10.798: INFO: csi-hostpath-resizer-0 started at 2019-11-11 21:14:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:10.798: INFO: Container csi-resizer ready: true, restart count 0 Nov 11 21:23:10.798: INFO: security-context-ea38c576-290b-4c9e-8bd0-6063b39e4673 started at 2019-11-11 21:22:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:10.798: INFO: Container write-pod ready: false, restart count 0 Nov 11 21:23:10.798: INFO: csi-hostpathplugin-0 started at 2019-11-11 21:23:02 +0000 UTC (0+0 container statuses recorded) Nov 11 21:23:10.798: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:20:33 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:10.798: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:23:10.798: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:20:38 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:10.798: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:23:10.798: INFO: local-client started at 2019-11-11 21:22:55 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:10.798: INFO: Container local-client ready: false, restart count 0 Nov 11 21:23:10.798: INFO: keyvault-flexvolume-zzp7p started at 2019-11-11 20:48:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:10.798: INFO: Container keyvault-flexvolume ready: true, restart count 0 Nov 11 21:23:10.798: INFO: csi-snapshotter-0 started at 2019-11-11 21:19:24 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:10.798: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 11 21:23:10.798: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:19:49 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:10.798: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:23:10.798: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:22:03 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:10.798: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:23:10.798: INFO: csi-hostpath-attacher-0 started at 2019-11-11 21:19:39 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:10.798: INFO: Container csi-attacher ready: true, restart count 0 Nov 11 21:23:10.798: INFO: csi-hostpath-provisioner-0 started at 2019-11-11 21:14:50 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:10.798: INFO: Container csi-provisioner ready: true, restart count 0 Nov 11 21:23:10.798: INFO: csi-hostpath-attacher-0 started at 2019-11-11 21:18:12 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:10.798: INFO: Container csi-attacher ready: true, restart count 0 Nov 11 21:23:10.798: INFO: kubernetes-dashboard-65966766b9-lv2x8 started at 2019-11-11 20:48:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:10.798: INFO: Container kubernetes-dashboard ready: true, restart count 0 Nov 11 21:23:10.798: INFO: csi-snapshotter-0 started at 2019-11-11 21:14:52 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:10.798: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 11 21:23:10.798: INFO: csi-hostpath-provisioner-0 started at 2019-11-11 21:19:42 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:10.798: INFO: Container csi-provisioner ready: true, restart count 0 Nov 11 21:23:10.798: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:21:34 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:10.798: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:23:10.798: INFO: pod-subpath-test-hostpathsymlink-pvr2 started at 2019-11-11 21:22:48 +0000 UTC (2+2 container statuses recorded) Nov 11 21:23:10.798: INFO: Init container init-volume-hostpathsymlink-pvr2 ready: true, restart count 0 Nov 11 21:23:10.798: INFO: Init container test-init-subpath-hostpathsymlink-pvr2 ready: false, restart count 0 Nov 11 21:23:10.798: INFO: Container test-container-subpath-hostpathsymlink-pvr2 ready: false, restart count 0 Nov 11 21:23:10.798: INFO: Container test-container-volume-hostpathsymlink-pvr2 ready: false, restart count 0 Nov 11 21:23:10.798: INFO: inline-volume-tester-v8qbq started at 2019-11-11 21:14:53 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:10.798: INFO: Container csi-volume-tester ready: false, restart count 0 Nov 11 21:23:10.798: INFO: csi-hostpathplugin-0 started at <nil> (0+0 container statuses recorded) Nov 11 21:23:10.798: INFO: csi-hostpath-resizer-0 started at 2019-11-11 21:19:42 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:10.798: INFO: Container csi-resizer ready: true, restart count 0 Nov 11 21:23:10.798: INFO: csi-hostpath-resizer-0 started at 2019-11-11 21:18:13 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:10.798: INFO: Container csi-resizer ready: true, restart count 0 Nov 11 21:23:10.798: INFO: csi-snapshotter-0 started at 2019-11-11 21:18:13 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:10.798: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 11 21:23:10.798: INFO: blobfuse-flexvol-installer-dmnhw started at 2019-11-11 20:48:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:10.798: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 Nov 11 21:23:10.798: INFO: csi-hostpath-provisioner-0 started at 2019-11-11 21:18:13 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:10.798: INFO: Container csi-provisioner ready: true, restart count 0 Nov 11 21:23:10.798: INFO: coredns-87f5d796-6mz2l started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:10.798: INFO: Container coredns ready: true, restart count 0 Nov 11 21:23:10.798: INFO: pod-subpath-test-local-preprovisionedpv-pkvl started at 2019-11-11 21:22:15 +0000 UTC (1+1 container statuses recorded) Nov 11 21:23:10.798: INFO: Init container init-volume-local-preprovisionedpv-pkvl ready: true, restart count 0 Nov 11 21:23:10.798: INFO: Container test-container-subpath-local-preprovisionedpv-pkvl ready: true, restart count 0 Nov 11 21:23:10.798: INFO: csi-hostpath-attacher-0 started at 2019-11-11 21:19:23 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:10.798: INFO: Container csi-attacher ready: true, restart count 0 Nov 11 21:23:10.798: INFO: pvc-datasource-tester-lm7kd started at 2019-11-11 21:22:49 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:10.798: INFO: Container volume-tester ready: false, restart count 0 Nov 11 21:23:10.798: INFO: local-client started at 2019-11-11 21:22:30 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:10.798: INFO: Container local-client ready: true, restart count 0 Nov 11 21:23:10.798: INFO: local-client started at 2019-11-11 21:22:54 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:10.798: INFO: Container local-client ready: false, restart count 0 Nov 11 21:23:10.798: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:22:41 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:10.798: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:23:10.798: INFO: csi-snapshotter-0 started at 2019-11-11 21:19:44 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:10.798: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 11 21:23:10.798: INFO: kube-proxy-nwdpz started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:10.798: INFO: Container kube-proxy ready: true, restart count 0 Nov 11 21:23:10.798: INFO: csi-hostpath-attacher-0 started at 2019-11-11 21:14:46 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:10.798: INFO: Container csi-attacher ready: true, restart count 0 Nov 11 21:23:10.798: INFO: csi-hostpath-provisioner-0 started at 2019-11-11 21:19:23 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:10.798: INFO: Container csi-provisioner ready: true, restart count 0 Nov 11 21:23:10.798: INFO: csi-hostpath-resizer-0 started at 2019-11-11 21:19:24 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:10.798: INFO: Container csi-resizer ready: true, restart count 0 Nov 11 21:23:10.798: INFO: csi-hostpathplugin-0 started at 2019-11-11 21:21:22 +0000 UTC (0+3 container statuses recorded) Nov 11 21:23:10.798: INFO: Container hostpath ready: true, restart count 0 Nov 11 21:23:10.798: INFO: Container liveness-probe ready: true, restart count 0 Nov 11 21:23:10.798: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 11 21:23:10.798: INFO: azure-ip-masq-agent-c4mb2 started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:10.798: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:23:10.798: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:22:36 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:10.798: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:23:10.798: INFO: csi-hostpathplugin-0 started at 2019-11-11 21:23:08 +0000 UTC (0+0 container statuses recorded) W1111 21:23:10.854585 93375 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:23:10.991: INFO: Latency metrics for node k8s-agentpool-35271922-vmss000000 Nov 11 21:23:10.991: INFO: Logging node info for node k8s-agentpool-35271922-vmss000001 Nov 11 21:23:11.047: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool-35271922-vmss000001 /api/v1/nodes/k8s-agentpool-35271922-vmss000001 af5ba0b6-4f7f-4468-9d3c-b3e116ca81ca 53101 0 2019-11-11 20:48:13 +0000 UTC <nil> <nil> map[agentpool:agentpool beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool-35271922-vmss000001 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: storageprofile:managed storagetier:Premium_LRS] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-8406":"k8s-agentpool-35271922-vmss000001","csi-hostpath-provisioning-2466":"k8s-agentpool-35271922-vmss000001","csi-hostpath-provisioning-265":"k8s-agentpool-35271922-vmss000001","csi-hostpath-provisioning-310":"k8s-agentpool-35271922-vmss000001","csi-hostpath-provisioning-753":"k8s-agentpool-35271922-vmss000001","csi-hostpath-volume-expand-3983":"k8s-agentpool-35271922-vmss000001","csi-mock-csi-mock-volumes-2781":"csi-mock-csi-mock-volumes-2781","csi-mock-csi-mock-volumes-2838":"csi-mock-csi-mock-volumes-2838","csi-mock-csi-mock-volumes-3306":"csi-mock-csi-mock-volumes-3306","csi-mock-csi-mock-volumes-6769":"csi-mock-csi-mock-volumes-6769","csi-mock-csi-mock-volumes-694":"csi-mock-csi-mock-volumes-694","csi-mock-csi-mock-volumes-797":"csi-mock-csi-mock-volumes-797"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-35271922-vmss/virtualMachines/1,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16797044736 0} {<nil>} 16403364Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16010612736 0} {<nil>} 15635364Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:22:30 +0000 UTC,LastTransitionTime:2019-11-11 20:48:13 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:22:30 +0000 UTC,LastTransitionTime:2019-11-11 20:48:13 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:22:30 +0000 UTC,LastTransitionTime:2019-11-11 20:48:13 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:22:30 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.248.0.5,},NodeAddress{Type:Hostname,Address:k8s-agentpool-35271922-vmss000001,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:1df8c22200b84e64b76150d232ecb929,SystemUUID:B03EE801-7A2D-7140-BE20-B2C844575A73,BootID:d32206f7-48b0-46c5-a542-39721566b456,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[quay.io/kubernetes_incubator/nfs-provisioner@sha256:df762117e3c891f2d2ddff46ecb0776ba1f9f3c44cfd7739b0683bcd7a7954a8 quay.io/kubernetes_incubator/nfs-provisioner:v2.2.2],SizeBytes:391772778,},ContainerImage{Names:[gcr.io/google-samples/gb-frontend@sha256:35cb427341429fac3df10ff74600ea73e8ec0754d78f9ce89e0b4f3d70d53ba6 gcr.io/google-samples/gb-frontend:v6],SizeBytes:373099368,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/volume/nfs@sha256:c2ad734346f608a5f7d69cfded93c4e8094069320657bd372d12ba21dea3ea71 gcr.io/kubernetes-e2e-test-images/volume/nfs:1.0],SizeBytes:225358913,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/jessie-dnsutils@sha256:ad583e33cb284f7ef046673809b146ec4053cda19b54a85d2b180a86169715eb gcr.io/kubernetes-e2e-test-images/jessie-dnsutils:1.0],SizeBytes:195659796,},ContainerImage{Names:[httpd@sha256:addd70e4ee83f3bc9a4c1c7c41e37927ba47faf639312fc936df3afad7926f5a httpd:2.4.39-alpine],SizeBytes:126894770,},ContainerImage{Names:[httpd@sha256:eb8ccf084cf3e80eece1add239effefd171eb39adbc154d33c14260d905d4060 httpd:2.4.38-alpine],SizeBytes:123781643,},ContainerImage{Names:[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0],SizeBytes:71821984,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727 gcr.io/kubernetes-e2e-test-images/agnhost:2.6],SizeBytes:57345321,},ContainerImage{Names:[quay.io/k8scsi/csi-provisioner@sha256:0efcb424f1dde9b9fb11a1a14f2e48ab47e1c3f08bc3a929990dcfcb1f7ab34f quay.io/k8scsi/csi-provisioner:v1.4.0-rc1],SizeBytes:54431016,},ContainerImage{Names:[quay.io/k8scsi/csi-snapshotter@sha256:e3d3e742e32d00488fdb401045b9b1d033d7ca0ab6e760f77b24750fc95e5f70 quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1],SizeBytes:51703561,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0],SizeBytes:46226754,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:0aba670b4d9d6b2e720bbf575d733156c676b693ca26501235444490300db838 quay.io/k8scsi/csi-attacher:v1.1.0],SizeBytes:42839085,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:7d46fb6eb8b890dc546029d1565d502b4a1d974d33625c6ee2bc7991b77fc1a1 quay.io/k8scsi/csi-resizer:v0.2.0],SizeBytes:42817100,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:f315c9042e56def3c05c6b04fe79ec9da6d39ddc557ca365a76cf35964ea08b6 quay.io/k8scsi/csi-resizer:v0.1.0],SizeBytes:42623056,},ContainerImage{Names:[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1],SizeBytes:42541759,},ContainerImage{Names:[gcr.io/google-containers/debian-base@sha256:6966a0aedd7592c18ff2dd803c08bd85780ee19f5e3a2e7cf908a4cd837afcde gcr.io/google-containers/debian-base:0.4.1],SizeBytes:42323657,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nonroot@sha256:d4ede5c74517090b6686219059118ed178cf4620f5db8781b32f806bb1e7395b gcr.io/kubernetes-e2e-test-images/nonroot:1.0],SizeBytes:42321438,},ContainerImage{Names:[redis@sha256:50899ea1ceed33fa03232f3ac57578a424faa1742c1ac9c7a7bdb95cdf19b858 redis:5.0.5-alpine],SizeBytes:29331594,},ContainerImage{Names:[quay.io/k8scsi/hostpathplugin@sha256:b4826e492fc1762fceaf9726f41575ca0a4567864d3d235da874818de18039de quay.io/k8scsi/hostpathplugin:v1.2.0-rc5],SizeBytes:28761497,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7],SizeBytes:22909487,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/echoserver@sha256:e9ba514b896cdf559eef8788b66c2c3ee55f3572df617647b4b0d8b6bf81cf19 gcr.io/kubernetes-e2e-test-images/echoserver:2.2],SizeBytes:21692741,},ContainerImage{Names:[quay.io/k8scsi/mock-driver@sha256:e0eed916b7d970bad2b7d9875f9ad16932f987f0f3d91ec5d86da68b0b5cc9d1 quay.io/k8scsi/mock-driver:v2.1.0],SizeBytes:16226335,},ContainerImage{Names:[nginx@sha256:485b610fefec7ff6c463ced9623314a04ed67e3945b9c08d7e53a47f6d108dc7 nginx:1.14-alpine],SizeBytes:16032814,},ContainerImage{Names:[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0],SizeBytes:15815995,},ContainerImage{Names:[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0],SizeBytes:14967303,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/dnsutils@sha256:2abeee84efb79c14d731966e034af33bf324d3b26ca28497555511ff094b3ddd gcr.io/kubernetes-e2e-test-images/dnsutils:1.1],SizeBytes:9349974,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nonewprivs@sha256:10066e9039219449fe3c81f38fe01928f87914150768ab81b62a468e51fa7411 gcr.io/kubernetes-e2e-test-images/nonewprivs:1.0],SizeBytes:6757579,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nautilus@sha256:33a732d4c42a266912a5091598a0f07653c9134db4b8d571690d8afd509e0bfc gcr.io/kubernetes-e2e-test-images/nautilus:1.0],SizeBytes:4753501,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/kitten@sha256:bcbc4875c982ab39aa7c4f6acf4a287f604e996d9f34a3fbda8c3d1a7457d1f6 gcr.io/kubernetes-e2e-test-images/kitten:1.0],SizeBytes:4747037,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/test-webserver@sha256:7f93d6e32798ff28bc6289254d0c2867fe2c849c8e46edc50f8624734309812e gcr.io/kubernetes-e2e-test-images/test-webserver:1.0],SizeBytes:4732240,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:4206620,},ContainerImage{Names:[alpine@sha256:8421d9a84432575381bfabd248f1eb56f3aa21d9d7cd2511583c68c9b7511d10 alpine:3.7],SizeBytes:4206494,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 gcr.io/kubernetes-e2e-test-images/mounttest:1.0],SizeBytes:1563521,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest-user@sha256:17319ca525ee003681fccf7e8c6b1b910ff4f49b653d939ac7f9b6e7c463933d gcr.io/kubernetes-e2e-test-images/mounttest-user:1.0],SizeBytes:1450451,},ContainerImage{Names:[busybox@sha256:1303dbf110c57f3edf68d9f5a16c082ec06c4cf7604831669faf2c712260b5a0 busybox:latest],SizeBytes:1219790,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29],SizeBytes:1154361,},ContainerImage{Names:[busybox@sha256:bbc3a03235220b170ba48a157dd097dd1379299370e1ed99ce976df0355d24f0 busybox:1.27],SizeBytes:1129289,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea k8s.gcr.io/pause-amd64:3.1 k8s.gcr.io/pause:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:23:11.048: INFO: Logging kubelet events for node k8s-agentpool-35271922-vmss000001 Nov 11 21:23:11.107: INFO: Logging pods the kubelet thinks is on node k8s-agentpool-35271922-vmss000001 Nov 11 21:23:11.167: INFO: logs-generator started at 2019-11-11 21:22:05 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:11.167: INFO: Container logs-generator ready: true, restart count 0 Nov 11 21:23:11.167: INFO: fail-once-local-txr4g started at 2019-11-11 21:21:56 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:11.167: INFO: Container c ready: false, restart count 1 Nov 11 21:23:11.167: INFO: metrics-server-58ff8c5ddf-thxvp started at 2019-11-11 20:48:53 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:11.167: INFO: Container metrics-server ready: true, restart count 0 Nov 11 21:23:11.167: INFO: test-webserver-6b218e8a-73be-4386-8364-595903f1edca started at 2019-11-11 21:22:48 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:11.167: INFO: Container test-webserver ready: false, restart count 0 Nov 11 21:23:11.167: INFO: fail-once-local-gwsx5 started at 2019-11-11 21:20:58 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:11.167: INFO: Container c ready: false, restart count 1 Nov 11 21:23:11.167: INFO: nfs-server started at 2019-11-11 21:18:43 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:11.167: INFO: Container nfs-server ready: true, restart count 0 Nov 11 21:23:11.167: INFO: client-containers-18817208-c75c-44ce-88e1-76c8590d0eec started at 2019-11-11 21:22:15 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:11.167: INFO: Container test-container ready: false, restart count 0 Nov 11 21:23:11.167: INFO: pod-submit-remove-4e755e27-38f7-4e59-a0f8-31aa18ae4fc4 started at 2019-11-11 21:22:09 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:11.168: INFO: Container nginx ready: true, restart count 0 Nov 11 21:23:11.168: INFO: backofflimit-jxgdp started at 2019-11-11 21:22:11 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:11.168: INFO: Container c ready: false, restart count 0 Nov 11 21:23:11.168: INFO: terminate-cmd-rpoff763d95d-70e1-4ce4-80b1-b33e29a44319 started at 2019-11-11 21:22:27 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:11.168: INFO: Container terminate-cmd-rpof ready: false, restart count 0 Nov 11 21:23:11.168: INFO: pfpod started at 2019-11-11 21:22:22 +0000 UTC (0+2 container statuses recorded) Nov 11 21:23:11.168: INFO: Container portforwardtester ready: false, restart count 0 Nov 11 21:23:11.168: INFO: Container readiness ready: false, restart count 0 Nov 11 21:23:11.168: INFO: keyvault-flexvolume-2s2wq started at 2019-11-11 20:48:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:11.168: INFO: Container keyvault-flexvolume ready: true, restart count 0 Nov 11 21:23:11.168: INFO: e2e-test-httpd-rc-6hxd6 started at 2019-11-11 21:22:06 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:11.168: INFO: Container e2e-test-httpd-rc ready: false, restart count 0 Nov 11 21:23:11.168: INFO: test-rollover-controller-lqzk7 started at 2019-11-11 21:22:47 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:11.168: INFO: Container httpd ready: false, restart count 0 Nov 11 21:23:11.168: INFO: fail-once-local-fqqbs started at 2019-11-11 21:22:13 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:11.168: INFO: Container c ready: false, restart count 0 Nov 11 21:23:11.168: INFO: kube-proxy-sgvk2 started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:11.168: INFO: Container kube-proxy ready: true, restart count 0 Nov 11 21:23:11.168: INFO: externalsvc-8rj4l started at 2019-11-11 21:22:45 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:11.168: INFO: Container externalsvc ready: false, restart count 0 Nov 11 21:23:11.168: INFO: e2e-test-httpd-rc-8fca2dbdbed1f8f0dbbf980d1f6929b5-l4sb6 started at 2019-11-11 21:22:08 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:11.168: INFO: Container e2e-test-httpd-rc ready: true, restart count 0 Nov 11 21:23:11.168: INFO: externalsvc-vfqmb started at 2019-11-11 21:22:46 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:11.168: INFO: Container externalsvc ready: false, restart count 0 Nov 11 21:23:11.168: INFO: azure-ip-masq-agent-9427k started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:11.168: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:23:11.168: INFO: sample-apiserver-deployment-8447597c78-w4btv started at 2019-11-11 21:22:17 +0000 UTC (0+2 container statuses recorded) Nov 11 21:23:11.168: INFO: Container etcd ready: false, restart count 0 Nov 11 21:23:11.168: INFO: Container sample-apiserver ready: false, restart count 0 Nov 11 21:23:11.168: INFO: sample-crd-conversion-webhook-deployment-64d485d9bb-qrlw8 started at 2019-11-11 21:22:37 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:11.168: INFO: Container sample-crd-conversion-webhook ready: false, restart count 0 Nov 11 21:23:11.168: INFO: fail-once-local-m96ld started at 2019-11-11 21:20:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:11.168: INFO: Container c ready: false, restart count 1 Nov 11 21:23:11.168: INFO: blobfuse-flexvol-installer-tvkwl started at 2019-11-11 20:48:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:11.168: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 Nov 11 21:23:11.168: INFO: tiller-deploy-7559b6b885-jsnn2 started at 2019-11-11 20:48:53 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:11.168: INFO: Container tiller ready: true, restart count 0 Nov 11 21:23:11.168: INFO: busybox-3cc5f96b-0409-4e72-b3ab-eae766eb08d2 started at 2019-11-11 21:22:14 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:11.168: INFO: Container busybox ready: false, restart count 0 Nov 11 21:23:11.168: INFO: image-pull-teste9cbad06-1466-4ada-adc6-8eae74913eff started at 2019-11-11 21:22:24 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:11.168: INFO: Container image-pull-test ready: false, restart count 0 W1111 21:23:11.222369 93375 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:23:12.214: INFO: Latency metrics for node k8s-agentpool-35271922-vmss000001 Nov 11 21:23:12.214: INFO: Logging node info for node k8s-master-35271922-vmss000000 Nov 11 21:23:12.284: INFO: Node Info: &Node{ObjectMeta:{k8s-master-35271922-vmss000000 /api/v1/nodes/k8s-master-35271922-vmss000000 f042d5b7-5ee8-4528-ae46-c997fef88c1c 53583 0 2019-11-11 20:48:07 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-35271922-vmss000000 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-35271922-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284363264 0} {<nil>} 7113636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6497931264 0} {<nil>} 6345636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:23:03 +0000 UTC,LastTransitionTime:2019-11-11 20:48:04 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:23:03 +0000 UTC,LastTransitionTime:2019-11-11 20:48:04 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:23:03 +0000 UTC,LastTransitionTime:2019-11-11 20:48:04 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:23:03 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.4,},NodeAddress{Type:Hostname,Address:k8s-master-35271922-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:8843d4f5cd0544d08ae501580db6a3a7,SystemUUID:41DCC090-746D-9E44-8888-8A77F0E0C51C,BootID:42d9eb02-ab49-4c2a-8791-7c5b2ac8f2bc,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:45e5b123e6188348715630acd2f7c1469c5b5181078ce90dc6f5399be4c17be1 k8sprow.azurecr.io/azure-cloud-controller-manager:1193984044413489154-9318bf1d],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:23:12.285: INFO: Logging kubelet events for node k8s-master-35271922-vmss000000 Nov 11 21:23:12.342: INFO: Logging pods the kubelet thinks is on node k8s-master-35271922-vmss000000 Nov 11 21:23:12.421: INFO: cloud-controller-manager-k8s-master-35271922-vmss000000 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:12.421: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 11 21:23:12.421: INFO: kube-addon-manager-k8s-master-35271922-vmss000000 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:12.421: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 11 21:23:12.421: INFO: kube-apiserver-k8s-master-35271922-vmss000000 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:12.421: INFO: Container kube-apiserver ready: true, restart count 0 Nov 11 21:23:12.421: INFO: kube-controller-manager-k8s-master-35271922-vmss000000 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:12.421: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 11 21:23:12.421: INFO: kube-scheduler-k8s-master-35271922-vmss000000 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:12.421: INFO: Container kube-scheduler ready: true, restart count 0 Nov 11 21:23:12.421: INFO: azure-ip-masq-agent-k7l5b started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:12.421: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:23:12.421: INFO: kube-proxy-8hzbt started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:12.421: INFO: Container kube-proxy ready: true, restart count 0 W1111 21:23:12.476452 93375 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:23:12.611: INFO: Latency metrics for node k8s-master-35271922-vmss000000 Nov 11 21:23:12.611: INFO: Logging node info for node k8s-master-35271922-vmss000001 Nov 11 21:23:12.664: INFO: Node Info: &Node{ObjectMeta:{k8s-master-35271922-vmss000001 /api/v1/nodes/k8s-master-35271922-vmss000001 0ff06487-2e4c-4c07-a6f2-d77d1a47bbcd 53606 0 2019-11-11 20:48:39 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-35271922-vmss000001 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.6.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-35271922-vmss/virtualMachines/1,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.6.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284363264 0} {<nil>} 7113636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6497931264 0} {<nil>} 6345636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:49:01 +0000 UTC,LastTransitionTime:2019-11-11 20:49:01 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:23:06 +0000 UTC,LastTransitionTime:2019-11-11 20:48:32 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:23:06 +0000 UTC,LastTransitionTime:2019-11-11 20:48:32 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:23:06 +0000 UTC,LastTransitionTime:2019-11-11 20:48:32 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:23:06 +0000 UTC,LastTransitionTime:2019-11-11 20:48:39 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.5,},NodeAddress{Type:Hostname,Address:k8s-master-35271922-vmss000001,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:24b17940193c48e38f59d10d619fb718,SystemUUID:25FDA250-5862-9148-98C8-17FFDD874AEC,BootID:cdd7987a-6ece-464d-a177-21f53ae0b68b,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:45e5b123e6188348715630acd2f7c1469c5b5181078ce90dc6f5399be4c17be1 k8sprow.azurecr.io/azure-cloud-controller-manager:1193984044413489154-9318bf1d],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:23:12.664: INFO: Logging kubelet events for node k8s-master-35271922-vmss000001 Nov 11 21:23:12.723: INFO: Logging pods the kubelet thinks is on node k8s-master-35271922-vmss000001 Nov 11 21:23:12.807: INFO: cloud-controller-manager-k8s-master-35271922-vmss000001 started at 2019-11-11 20:48:26 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:12.807: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 11 21:23:12.807: INFO: kube-addon-manager-k8s-master-35271922-vmss000001 started at 2019-11-11 20:48:26 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:12.807: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 11 21:23:12.807: INFO: kube-apiserver-k8s-master-35271922-vmss000001 started at 2019-11-11 20:48:26 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:12.807: INFO: Container kube-apiserver ready: true, restart count 0 Nov 11 21:23:12.807: INFO: kube-controller-manager-k8s-master-35271922-vmss000001 started at 2019-11-11 20:48:26 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:12.807: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 11 21:23:12.807: INFO: kube-scheduler-k8s-master-35271922-vmss000001 started at 2019-11-11 20:48:26 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:12.807: INFO: Container kube-scheduler ready: true, restart count 0 Nov 11 21:23:12.807: INFO: azure-ip-masq-agent-bm2t5 started at 2019-11-11 20:48:40 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:12.807: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:23:12.807: INFO: kube-proxy-5fcm6 started at 2019-11-11 20:48:40 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:12.807: INFO: Container kube-proxy ready: true, restart count 0 W1111 21:23:12.861019 93375 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:23:13.038: INFO: Latency metrics for node k8s-master-35271922-vmss000001 Nov 11 21:23:13.038: INFO: Logging node info for node k8s-master-35271922-vmss000002 Nov 11 21:23:13.091: INFO: Node Info: &Node{ObjectMeta:{k8s-master-35271922-vmss000002 /api/v1/nodes/k8s-master-35271922-vmss000002 7fedbabf-dce6-48d1-994b-628369e7bcb5 53598 0 2019-11-11 20:48:08 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-35271922-vmss000002 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.4.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-35271922-vmss/virtualMachines/2,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284363264 0} {<nil>} 7113636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6497931264 0} {<nil>} 6345636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:23:05 +0000 UTC,LastTransitionTime:2019-11-11 20:48:03 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:23:05 +0000 UTC,LastTransitionTime:2019-11-11 20:48:03 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:23:05 +0000 UTC,LastTransitionTime:2019-11-11 20:48:03 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:23:05 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.6,},NodeAddress{Type:Hostname,Address:k8s-master-35271922-vmss000002,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:5ecd4c723b8845b18d85d341ba062204,SystemUUID:9E772AEA-D3D7-084D-98ED-1C1A511168B4,BootID:a73a4655-81c3-42e2-9d20-ef7e3cff3664,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:45e5b123e6188348715630acd2f7c1469c5b5181078ce90dc6f5399be4c17be1 k8sprow.azurecr.io/azure-cloud-controller-manager:1193984044413489154-9318bf1d],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:23:13.091: INFO: Logging kubelet events for node k8s-master-35271922-vmss000002 Nov 11 21:23:13.148: INFO: Logging pods the kubelet thinks is on node k8s-master-35271922-vmss000002 Nov 11 21:23:13.227: INFO: cloud-controller-manager-k8s-master-35271922-vmss000002 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:13.227: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 11 21:23:13.227: INFO: kube-addon-manager-k8s-master-35271922-vmss000002 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:13.227: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 11 21:23:13.227: INFO: kube-apiserver-k8s-master-35271922-vmss000002 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:13.227: INFO: Container kube-apiserver ready: true, restart count 0 Nov 11 21:23:13.227: INFO: kube-controller-manager-k8s-master-35271922-vmss000002 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:13.227: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 11 21:23:13.227: INFO: kube-scheduler-k8s-master-35271922-vmss000002 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:13.227: INFO: Container kube-scheduler ready: true, restart count 0 Nov 11 21:23:13.227: INFO: azure-ip-masq-agent-cmbqm started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:13.227: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:23:13.227: INFO: kube-proxy-t2sp7 started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:13.227: INFO: Container kube-proxy ready: true, restart count 0 W1111 21:23:13.281926 93375 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:23:13.411: INFO: Latency metrics for node k8s-master-35271922-vmss000002 Nov 11 21:23:13.411: INFO: Logging node info for node k8s-master-35271922-vmss000003 Nov 11 21:23:13.464: INFO: Node Info: &Node{ObjectMeta:{k8s-master-35271922-vmss000003 /api/v1/nodes/k8s-master-35271922-vmss000003 d4d04c51-fc7b-4cdc-a1e3-69990a198568 53603 0 2019-11-11 20:48:10 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-35271922-vmss000003 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.5.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-35271922-vmss/virtualMachines/3,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.5.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284363264 0} {<nil>} 7113636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6497931264 0} {<nil>} 6345636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:23:06 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:23:06 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:23:06 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:23:06 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.7,},NodeAddress{Type:Hostname,Address:k8s-master-35271922-vmss000003,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:53117d11ac104f35951d19e4a4448f8f,SystemUUID:048B9870-F8AE-BC47-80D8-26703FED9760,BootID:fc64edd8-d9c5-4d6f-b9e7-c902146bc724,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:45e5b123e6188348715630acd2f7c1469c5b5181078ce90dc6f5399be4c17be1 k8sprow.azurecr.io/azure-cloud-controller-manager:1193984044413489154-9318bf1d],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:23:13.465: INFO: Logging kubelet events for node k8s-master-35271922-vmss000003 Nov 11 21:23:13.521: INFO: Logging pods the kubelet thinks is on node k8s-master-35271922-vmss000003 Nov 11 21:23:13.602: INFO: cloud-controller-manager-k8s-master-35271922-vmss000003 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:13.602: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 11 21:23:13.602: INFO: kube-addon-manager-k8s-master-35271922-vmss000003 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:13.602: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 11 21:23:13.602: INFO: kube-apiserver-k8s-master-35271922-vmss000003 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:13.602: INFO: Container kube-apiserver ready: true, restart count 0 Nov 11 21:23:13.602: INFO: kube-controller-manager-k8s-master-35271922-vmss000003 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:13.602: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 11 21:23:13.602: INFO: kube-scheduler-k8s-master-35271922-vmss000003 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:13.602: INFO: Container kube-scheduler ready: true, restart count 0 Nov 11 21:23:13.602: INFO: azure-ip-masq-agent-rs668 started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:13.602: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:23:13.602: INFO: kube-proxy-tqjkf started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:13.602: INFO: Container kube-proxy ready: true, restart count 0 W1111 21:23:13.657674 93375 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:23:13.777: INFO: Latency metrics for node k8s-master-35271922-vmss000003 Nov 11 21:23:13.778: INFO: Logging node info for node k8s-master-35271922-vmss000004 Nov 11 21:23:13.832: INFO: Node Info: &Node{ObjectMeta:{k8s-master-35271922-vmss000004 /api/v1/nodes/k8s-master-35271922-vmss000004 edf29ad2-2fc8-4411-ac05-e1bf8781e340 53617 0 2019-11-11 20:48:11 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-35271922-vmss000004 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-35271922-vmss/virtualMachines/4,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284363264 0} {<nil>} 7113636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6497931264 0} {<nil>} 6345636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:23:07 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:23:07 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:23:07 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:23:07 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.8,},NodeAddress{Type:Hostname,Address:k8s-master-35271922-vmss000004,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:3ecc95ca67e14e2e87659efc6707e3b4,SystemUUID:0649BB2F-7C03-5743-A309-9BABFF3DD11A,BootID:1256b8d0-bf8e-4490-a45d-beaf41e45a09,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:45e5b123e6188348715630acd2f7c1469c5b5181078ce90dc6f5399be4c17be1 k8sprow.azurecr.io/azure-cloud-controller-manager:1193984044413489154-9318bf1d],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:23:13.832: INFO: Logging kubelet events for node k8s-master-35271922-vmss000004 Nov 11 21:23:13.888: INFO: Logging pods the kubelet thinks is on node k8s-master-35271922-vmss000004 Nov 11 21:23:13.965: INFO: kube-controller-manager-k8s-master-35271922-vmss000004 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:13.965: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 11 21:23:13.965: INFO: kube-scheduler-k8s-master-35271922-vmss000004 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:13.965: INFO: Container kube-scheduler ready: true, restart count 0 Nov 11 21:23:13.965: INFO: azure-ip-masq-agent-trzsm started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:13.965: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:23:13.965: INFO: kube-proxy-zrmqq started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:13.965: INFO: Container kube-proxy ready: true, restart count 0 Nov 11 21:23:13.965: INFO: cloud-controller-manager-k8s-master-35271922-vmss000004 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:13.965: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 11 21:23:13.965: INFO: kube-addon-manager-k8s-master-35271922-vmss000004 started at 2019-11-11 20:48:19 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:13.965: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 11 21:23:13.965: INFO: kube-apiserver-k8s-master-35271922-vmss000004 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:23:13.965: INFO: Container kube-apiserver ready: true, restart count 0 W1111 21:23:14.024489 93375 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:23:14.166: INFO: Latency metrics for node k8s-master-35271922-vmss000004 Nov 11 21:23:14.166: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "volume-8245" for this suite. Nov 11 21:25:16.397: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered Nov 11 21:25:18.275: INFO: namespace volume-8245 deletion completed in 2m4.055385635s
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[sig\-storage\]\sCSI\sVolumes\s\[Driver\:\scsi\-hostpath\]\s\[Testpattern\:\sDynamic\sPV\s\(default\sfs\)\]\ssubPath\sshould\ssupport\sexisting\sdirectories\swhen\sreadOnly\sspecified\sin\sthe\svolumeSource$'
test/e2e/storage/testsuites/subpath.go:377 Nov 11 21:12:59.399: Unexpected error: <*errors.errorString | 0xc001f19c90>: { s: "expected pod \"pod-subpath-test-csi-hostpath-dynamicpv-ns5x\" success: Gave up after waiting 5m0s for pod \"pod-subpath-test-csi-hostpath-dynamicpv-ns5x\" to be \"success or failure\"", } expected pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x" success: Gave up after waiting 5m0s for pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x" to be "success or failure" occurred test/e2e/framework/util.go:1667from junit_10.xml
[BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath test/e2e/storage/testsuites/base.go:93 [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath test/e2e/framework/framework.go:151 �[1mSTEP�[0m: Creating a kubernetes client Nov 11 21:02:35.276: INFO: >>> kubeConfig: /workspace/aks147280421/kubeconfig/kubeconfig.westus2.json �[1mSTEP�[0m: Building a namespace api object, basename provisioning �[1mSTEP�[0m: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in provisioning-4690 �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace [It] should support existing directories when readOnly specified in the volumeSource test/e2e/storage/testsuites/subpath.go:377 �[1mSTEP�[0m: deploying csi-hostpath driver Nov 11 21:02:35.892: INFO: creating *v1.ServiceAccount: provisioning-4690/csi-attacher Nov 11 21:02:35.955: INFO: creating *v1.ClusterRole: external-attacher-runner-provisioning-4690 Nov 11 21:02:35.955: INFO: Define cluster role external-attacher-runner-provisioning-4690 Nov 11 21:02:36.021: INFO: creating *v1.ClusterRoleBinding: csi-attacher-role-provisioning-4690 Nov 11 21:02:36.085: INFO: creating *v1.Role: provisioning-4690/external-attacher-cfg-provisioning-4690 Nov 11 21:02:36.153: INFO: creating *v1.RoleBinding: provisioning-4690/csi-attacher-role-cfg Nov 11 21:02:36.215: INFO: creating *v1.ServiceAccount: provisioning-4690/csi-provisioner Nov 11 21:02:36.275: INFO: creating *v1.ClusterRole: external-provisioner-runner-provisioning-4690 Nov 11 21:02:36.275: INFO: Define cluster role external-provisioner-runner-provisioning-4690 Nov 11 21:02:36.335: INFO: creating *v1.ClusterRoleBinding: csi-provisioner-role-provisioning-4690 Nov 11 21:02:36.398: INFO: creating *v1.Role: provisioning-4690/external-provisioner-cfg-provisioning-4690 Nov 11 21:02:36.465: INFO: creating *v1.RoleBinding: provisioning-4690/csi-provisioner-role-cfg Nov 11 21:02:36.525: INFO: creating *v1.ServiceAccount: provisioning-4690/csi-snapshotter Nov 11 21:02:36.590: INFO: creating *v1.ClusterRole: external-snapshotter-runner-provisioning-4690 Nov 11 21:02:36.590: INFO: Define cluster role external-snapshotter-runner-provisioning-4690 Nov 11 21:02:36.649: INFO: creating *v1.ClusterRoleBinding: csi-snapshotter-role-provisioning-4690 Nov 11 21:02:36.724: INFO: creating *v1.Role: provisioning-4690/external-snapshotter-leaderelection-provisioning-4690 Nov 11 21:02:36.783: INFO: creating *v1.RoleBinding: provisioning-4690/external-snapshotter-leaderelection Nov 11 21:02:36.843: INFO: creating *v1.ServiceAccount: provisioning-4690/csi-resizer Nov 11 21:02:36.904: INFO: creating *v1.ClusterRole: external-resizer-runner-provisioning-4690 Nov 11 21:02:36.905: INFO: Define cluster role external-resizer-runner-provisioning-4690 Nov 11 21:02:36.997: INFO: creating *v1.ClusterRoleBinding: csi-resizer-role-provisioning-4690 Nov 11 21:02:37.060: INFO: creating *v1.Role: provisioning-4690/external-resizer-cfg-provisioning-4690 Nov 11 21:02:37.126: INFO: creating *v1.RoleBinding: provisioning-4690/csi-resizer-role-cfg Nov 11 21:02:37.186: INFO: creating *v1.Service: provisioning-4690/csi-hostpath-attacher Nov 11 21:02:37.274: INFO: creating *v1.StatefulSet: provisioning-4690/csi-hostpath-attacher Nov 11 21:02:37.339: INFO: creating *v1beta1.CSIDriver: csi-hostpath-provisioning-4690 Nov 11 21:02:37.399: INFO: creating *v1.Service: provisioning-4690/csi-hostpathplugin Nov 11 21:02:37.488: INFO: creating *v1.StatefulSet: provisioning-4690/csi-hostpathplugin Nov 11 21:02:37.569: INFO: creating *v1.Service: provisioning-4690/csi-hostpath-provisioner Nov 11 21:02:37.648: INFO: creating *v1.StatefulSet: provisioning-4690/csi-hostpath-provisioner Nov 11 21:02:37.724: INFO: creating *v1.Service: provisioning-4690/csi-hostpath-resizer Nov 11 21:02:37.800: INFO: creating *v1.StatefulSet: provisioning-4690/csi-hostpath-resizer Nov 11 21:02:37.866: INFO: creating *v1.Service: provisioning-4690/csi-snapshotter Nov 11 21:02:37.961: INFO: creating *v1.StatefulSet: provisioning-4690/csi-snapshotter Nov 11 21:02:38.026: INFO: creating *v1.ClusterRoleBinding: psp-csi-hostpath-role-provisioning-4690 Nov 11 21:02:38.092: INFO: Test running for native CSI Driver, not checking metrics Nov 11 21:02:38.092: INFO: Creating resource for dynamic PV �[1mSTEP�[0m: creating a StorageClass provisioning-4690-csi-hostpath-provisioning-4690-scsf8l7 �[1mSTEP�[0m: creating a claim Nov 11 21:02:38.169: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil Nov 11 21:02:38.234: INFO: Waiting up to 5m0s for PersistentVolumeClaims [csi-hostpathhx7qn] to have phase Bound Nov 11 21:02:38.300: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:02:40.357: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:02:42.414: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:02:44.470: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:02:46.527: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:02:48.583: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:02:50.640: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:02:52.697: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:02:54.754: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:02:56.810: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:02:58.870: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:03:00.927: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:03:02.984: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:03:05.041: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:03:07.099: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:03:09.155: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:03:11.216: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:03:13.272: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:03:15.328: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:03:17.388: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:03:19.452: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:03:21.508: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:03:23.565: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:03:25.628: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:03:27.691: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:03:29.749: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:03:31.806: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:03:33.862: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:03:35.919: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:03:37.989: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:03:40.046: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:03:42.103: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:03:44.160: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:03:46.216: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:03:48.273: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:03:50.330: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:03:52.388: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:03:54.448: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:03:56.505: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:03:58.561: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:04:00.620: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:04:02.683: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:04:04.740: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:04:06.797: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:04:08.854: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:04:10.910: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:04:12.968: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:04:15.025: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:04:17.106: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:04:19.163: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:04:21.222: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:04:23.282: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:04:25.340: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:04:27.404: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:04:29.480: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:04:31.536: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:04:33.594: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:04:35.650: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:04:37.711: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:04:39.768: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:04:41.827: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:04:43.884: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:04:45.944: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:04:48.002: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:04:50.060: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:04:52.122: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:04:54.179: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:04:56.236: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:04:58.293: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:05:00.349: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:05:02.409: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:05:04.465: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:05:06.528: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:05:08.585: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:05:10.642: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:05:12.707: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:05:14.764: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:05:16.823: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:05:18.879: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:05:20.936: INFO: PersistentVolumeClaim csi-hostpathhx7qn found but phase is Pending instead of Bound. Nov 11 21:05:22.994: INFO: PersistentVolumeClaim csi-hostpathhx7qn found and phase=Bound (2m44.759252501s) �[1mSTEP�[0m: Creating pod pod-subpath-test-csi-hostpath-dynamicpv-ns5x �[1mSTEP�[0m: Creating a pod to test subpath Nov 11 21:05:23.182: INFO: Waiting up to 5m0s for pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x" in namespace "provisioning-4690" to be "success or failure" Nov 11 21:05:23.239: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 56.560385ms Nov 11 21:05:25.296: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 2.113932542s Nov 11 21:05:27.353: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 4.170978591s Nov 11 21:05:29.457: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 6.274579974s Nov 11 21:05:31.515: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 8.332513673s Nov 11 21:05:33.573: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 10.390381995s Nov 11 21:05:35.634: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 12.451669205s Nov 11 21:05:37.691: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 14.509096078s Nov 11 21:05:39.748: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 16.565813481s Nov 11 21:05:41.806: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 18.62348766s Nov 11 21:05:43.863: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 20.681035281s Nov 11 21:05:45.920: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 22.737783311s Nov 11 21:05:47.977: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 24.795305588s Nov 11 21:05:50.035: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 26.852686375s Nov 11 21:05:52.092: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 28.909662399s Nov 11 21:05:54.150: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 30.968153255s Nov 11 21:05:56.208: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 33.025957713s Nov 11 21:05:58.268: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 35.085781974s Nov 11 21:06:00.325: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 37.142652828s Nov 11 21:06:02.381: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 39.198807619s Nov 11 21:06:04.438: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 41.255618537s Nov 11 21:06:06.495: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 43.312627433s Nov 11 21:06:08.559: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 45.376578155s Nov 11 21:06:10.617: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 47.434558767s Nov 11 21:06:12.674: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 49.492085617s Nov 11 21:06:14.732: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 51.549566588s Nov 11 21:06:16.793: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 53.610524345s Nov 11 21:06:18.850: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 55.66776819s Nov 11 21:06:20.908: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 57.725575263s Nov 11 21:06:22.965: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 59.78277194s Nov 11 21:06:25.022: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m1.83981955s Nov 11 21:06:27.079: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m3.897006124s Nov 11 21:06:29.136: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m5.954211587s Nov 11 21:06:31.194: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m8.011333008s Nov 11 21:06:33.251: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m10.068768666s Nov 11 21:06:35.309: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m12.126564138s Nov 11 21:06:37.366: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m14.183956263s Nov 11 21:06:39.424: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m16.241523589s Nov 11 21:06:41.483: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m18.300796439s Nov 11 21:06:43.554: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m20.371460179s Nov 11 21:06:45.612: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m22.42939114s Nov 11 21:06:47.671: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m24.488486118s Nov 11 21:06:49.728: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m26.545514451s Nov 11 21:06:51.786: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m28.603942213s Nov 11 21:06:53.844: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m30.661453265s Nov 11 21:06:55.902: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m32.719638776s Nov 11 21:06:57.961: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m34.778645076s Nov 11 21:07:00.018: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m36.836324766s Nov 11 21:07:02.080: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m38.897864524s Nov 11 21:07:04.141: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m40.958673701s Nov 11 21:07:06.200: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m43.018035859s Nov 11 21:07:08.260: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m45.077400819s Nov 11 21:07:10.318: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m47.135732547s Nov 11 21:07:12.376: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m49.193624819s Nov 11 21:07:14.436: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m51.253729405s Nov 11 21:07:16.494: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m53.311568965s Nov 11 21:07:18.558: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m55.375599412s Nov 11 21:07:20.618: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m57.435646813s Nov 11 21:07:22.677: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m59.494885666s Nov 11 21:07:24.736: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 2m1.553599667s Nov 11 21:07:26.793: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 2m3.610458379s Nov 11 21:07:28.851: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 2m5.668605896s Nov 11 21:07:30.911: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 2m7.728430219s Nov 11 21:07:33.018: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 2m9.835604862s Nov 11 21:07:35.076: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 2m11.893475009s Nov 11 21:07:37.133: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 2m13.950621509s Nov 11 21:07:39.191: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 2m16.008761238s Nov 11 21:07:41.249: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 2m18.066423586s Nov 11 21:07:43.306: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 2m20.123580672s Nov 11 21:07:45.363: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 2m22.181241149s Nov 11 21:07:47.422: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Succeeded", Reason="", readiness=false. Elapsed: 2m24.239700833s �[1mSTEP�[0m: Saw pod success Nov 11 21:07:47.422: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x" satisfied condition "success or failure" Nov 11 21:07:47.479: INFO: Trying to get logs from node k8s-agentpool-35271922-vmss000000 pod pod-subpath-test-csi-hostpath-dynamicpv-ns5x container test-container-subpath-csi-hostpath-dynamicpv-ns5x: <nil> �[1mSTEP�[0m: delete the pod Nov 11 21:07:47.623: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-ns5x to disappear Nov 11 21:07:47.681: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-ns5x no longer exists �[1mSTEP�[0m: Deleting pod pod-subpath-test-csi-hostpath-dynamicpv-ns5x Nov 11 21:07:47.681: INFO: Deleting pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x" in namespace "provisioning-4690" �[1mSTEP�[0m: Creating pod pod-subpath-test-csi-hostpath-dynamicpv-ns5x �[1mSTEP�[0m: Creating a pod to test subpath Nov 11 21:07:47.813: INFO: Waiting up to 5m0s for pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x" in namespace "provisioning-4690" to be "success or failure" Nov 11 21:07:47.868: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 55.697365ms Nov 11 21:07:49.926: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 2.113006302s Nov 11 21:07:51.987: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 4.174094581s Nov 11 21:07:54.047: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 6.234690949s Nov 11 21:07:56.106: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 8.293362432s Nov 11 21:07:58.163: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 10.350359675s Nov 11 21:08:00.226: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 12.412961888s Nov 11 21:08:02.292: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 14.478907938s Nov 11 21:08:04.351: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 16.538005972s Nov 11 21:08:06.410: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 18.59761204s Nov 11 21:08:08.564: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 20.750783449s Nov 11 21:08:10.633: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 22.819995939s Nov 11 21:08:12.693: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 24.879985485s Nov 11 21:08:14.751: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 26.938074387s Nov 11 21:08:16.809: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 28.996320481s Nov 11 21:08:18.866: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 31.053743662s Nov 11 21:08:20.932: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 33.119246089s Nov 11 21:08:22.989: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 35.176457849s Nov 11 21:08:25.082: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 37.269009065s Nov 11 21:08:27.143: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 39.330124044s Nov 11 21:08:29.201: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 41.387957561s Nov 11 21:08:31.396: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 43.583065028s Nov 11 21:08:33.456: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 45.642801905s Nov 11 21:08:35.513: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 47.700022623s Nov 11 21:08:37.578: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 49.765632066s Nov 11 21:08:39.636: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 51.822817345s Nov 11 21:08:41.694: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 53.881253997s Nov 11 21:08:43.767: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 55.954214887s Nov 11 21:08:45.825: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 58.011958318s Nov 11 21:08:47.881: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m0.068751629s Nov 11 21:08:49.938: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m2.125511325s Nov 11 21:08:51.997: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m4.183789266s Nov 11 21:08:54.058: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m6.245485151s Nov 11 21:08:56.116: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m8.302987437s Nov 11 21:08:58.175: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m10.361957347s Nov 11 21:09:00.232: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m12.419043601s Nov 11 21:09:02.291: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m14.478103859s Nov 11 21:09:04.360: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m16.547299455s Nov 11 21:09:06.418: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m18.605363479s Nov 11 21:09:08.476: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m20.66312354s Nov 11 21:09:10.566: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m22.753755637s Nov 11 21:09:12.625: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m24.812088653s Nov 11 21:09:14.698: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m26.885504711s Nov 11 21:09:16.756: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m28.942904516s Nov 11 21:09:18.815: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m31.002077148s Nov 11 21:09:20.872: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m33.058942291s Nov 11 21:09:22.928: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m35.115722795s Nov 11 21:09:25.022: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m37.208856371s Nov 11 21:09:27.078: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m39.265432916s Nov 11 21:09:29.137: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m41.323926007s Nov 11 21:09:31.194: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m43.381395419s Nov 11 21:09:33.252: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m45.439078448s Nov 11 21:09:35.315: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m47.502657694s Nov 11 21:09:37.374: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m49.56136418s Nov 11 21:09:39.431: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m51.618751583s Nov 11 21:09:41.489: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m53.676619168s Nov 11 21:09:43.561: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m55.748068776s Nov 11 21:09:45.623: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m57.809905352s Nov 11 21:09:47.681: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m59.867900995s Nov 11 21:09:49.738: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 2m1.925726875s Nov 11 21:09:51.796: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 2m3.982967388s Nov 11 21:09:53.977: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 2m6.164153839s Nov 11 21:09:56.035: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 2m8.221879617s Nov 11 21:09:58.112: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 2m10.298917636s Nov 11 21:10:00.170: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 2m12.357610164s Nov 11 21:10:02.227: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 2m14.414058252s Nov 11 21:10:04.285: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 2m16.472274519s Nov 11 21:10:06.344: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 2m18.53102599s Nov 11 21:10:08.402: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 2m20.589471555s Nov 11 21:10:10.460: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 2m22.647679305s Nov 11 21:10:12.522: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 2m24.709113672s Nov 11 21:10:14.579: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 2m26.76621108s Nov 11 21:10:16.643: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 2m28.829956296s Nov 11 21:10:18.702: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 2m30.888850129s Nov 11 21:10:20.760: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 2m32.947304026s Nov 11 21:10:22.817: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 2m35.003868292s Nov 11 21:10:24.875: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 2m37.061869839s Nov 11 21:10:26.934: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 2m39.120797375s Nov 11 21:10:29.017: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 2m41.203948857s Nov 11 21:10:31.077: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 2m43.263973825s Nov 11 21:10:33.133: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 2m45.32018977s Nov 11 21:10:35.189: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 2m47.376617396s Nov 11 21:10:37.246: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 2m49.43337948s Nov 11 21:10:39.305: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 2m51.491979165s Nov 11 21:10:41.362: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 2m53.549107205s Nov 11 21:10:43.419: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 2m55.606766767s Nov 11 21:10:45.478: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 2m57.665158973s Nov 11 21:10:47.535: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 2m59.721825329s Nov 11 21:10:49.592: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 3m1.779031061s Nov 11 21:10:51.651: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 3m3.837937621s Nov 11 21:10:53.711: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 3m5.897947887s Nov 11 21:10:55.769: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 3m7.956100419s Nov 11 21:10:57.826: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 3m10.013067026s Nov 11 21:10:59.885: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 3m12.072201669s Nov 11 21:11:01.943: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 3m14.130175982s Nov 11 21:11:04.001: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 3m16.188448953s Nov 11 21:11:06.059: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 3m18.246225427s Nov 11 21:11:08.116: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 3m20.303089575s Nov 11 21:11:10.187: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 3m22.373793512s Nov 11 21:11:12.246: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 3m24.433754435s Nov 11 21:11:14.304: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 3m26.491216773s Nov 11 21:11:16.362: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 3m28.548838318s Nov 11 21:11:18.418: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 3m30.605464285s Nov 11 21:11:20.475: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 3m32.662540457s Nov 11 21:11:22.535: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 3m34.72228978s Nov 11 21:11:24.594: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 3m36.781520688s Nov 11 21:11:26.651: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 3m38.838017449s Nov 11 21:11:28.708: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 3m40.895392656s Nov 11 21:11:30.765: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 3m42.952151028s Nov 11 21:11:32.824: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 3m45.01146021s Nov 11 21:11:34.888: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 3m47.074901986s Nov 11 21:11:36.946: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 3m49.132823339s Nov 11 21:11:39.003: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 3m51.190174188s Nov 11 21:11:41.063: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 3m53.250728052s Nov 11 21:11:43.124: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 3m55.311239028s Nov 11 21:11:45.183: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 3m57.370507341s Nov 11 21:11:47.240: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 3m59.42701306s Nov 11 21:11:49.297: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 4m1.484706821s Nov 11 21:11:51.354: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 4m3.541001151s Nov 11 21:11:53.449: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 4m5.636500924s Nov 11 21:11:55.506: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 4m7.693105798s Nov 11 21:11:57.570: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 4m9.756791559s Nov 11 21:11:59.626: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 4m11.813430578s Nov 11 21:12:01.716: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 4m13.903340229s Nov 11 21:12:03.775: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 4m15.961939599s Nov 11 21:12:05.836: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 4m18.023425054s Nov 11 21:12:07.893: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 4m20.080372776s Nov 11 21:12:09.950: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 4m22.137004433s Nov 11 21:12:12.015: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 4m24.201920803s Nov 11 21:12:14.072: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 4m26.258960354s Nov 11 21:12:16.133: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 4m28.320604794s Nov 11 21:12:18.191: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 4m30.378086503s Nov 11 21:12:20.248: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 4m32.435232748s Nov 11 21:12:22.307: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 4m34.494628559s Nov 11 21:12:24.368: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 4m36.555755629s Nov 11 21:12:26.425: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 4m38.612460626s Nov 11 21:12:28.483: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 4m40.670115574s Nov 11 21:12:30.549: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 4m42.735907599s Nov 11 21:12:32.609: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 4m44.796532703s Nov 11 21:12:34.666: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 4m46.853661349s Nov 11 21:12:36.723: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 4m48.909983522s Nov 11 21:12:38.780: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 4m50.967448833s Nov 11 21:12:40.837: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 4m53.024234674s Nov 11 21:12:42.894: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 4m55.081244441s Nov 11 21:12:44.951: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 4m57.137980629s Nov 11 21:12:47.011: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x": Phase="Pending", Reason="", readiness=false. Elapsed: 4m59.198675998s Nov 11 21:12:49.154: INFO: Failed to get logs from node "k8s-agentpool-35271922-vmss000000" pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x" container "test-container-subpath-csi-hostpath-dynamicpv-ns5x": the server rejected our request for an unknown reason (get pods pod-subpath-test-csi-hostpath-dynamicpv-ns5x) �[1mSTEP�[0m: delete the pod Nov 11 21:12:49.221: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-ns5x to disappear Nov 11 21:12:49.338: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-ns5x still exists Nov 11 21:12:51.339: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-ns5x to disappear Nov 11 21:12:51.398: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-ns5x still exists Nov 11 21:12:53.339: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-ns5x to disappear Nov 11 21:12:53.448: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-ns5x still exists Nov 11 21:12:55.339: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-ns5x to disappear Nov 11 21:12:55.449: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-ns5x still exists Nov 11 21:12:57.339: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-ns5x to disappear Nov 11 21:12:57.449: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-ns5x still exists Nov 11 21:12:59.339: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-ns5x to disappear Nov 11 21:12:59.398: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-ns5x no longer exists Nov 11 21:12:59.399: FAIL: Unexpected error: <*errors.errorString | 0xc001f19c90>: { s: "expected pod \"pod-subpath-test-csi-hostpath-dynamicpv-ns5x\" success: Gave up after waiting 5m0s for pod \"pod-subpath-test-csi-hostpath-dynamicpv-ns5x\" to be \"success or failure\"", } expected pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x" success: Gave up after waiting 5m0s for pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x" to be "success or failure" occurred �[1mSTEP�[0m: Deleting pod Nov 11 21:12:59.399: INFO: Deleting pod "pod-subpath-test-csi-hostpath-dynamicpv-ns5x" in namespace "provisioning-4690" �[1mSTEP�[0m: Deleting pvc Nov 11 21:12:59.465: INFO: Deleting PersistentVolumeClaim "csi-hostpathhx7qn" Nov 11 21:12:59.532: INFO: Waiting up to 5m0s for PersistentVolume pvc-525180af-f06e-45d6-b287-04757cf4b323 to get deleted Nov 11 21:12:59.595: INFO: PersistentVolume pvc-525180af-f06e-45d6-b287-04757cf4b323 found and phase=Released (63.071906ms) Nov 11 21:13:04.651: INFO: PersistentVolume pvc-525180af-f06e-45d6-b287-04757cf4b323 was removed �[1mSTEP�[0m: Deleting sc �[1mSTEP�[0m: uninstalling csi-hostpath driver Nov 11 21:13:04.716: INFO: deleting *v1.ServiceAccount: provisioning-4690/csi-attacher Nov 11 21:13:04.784: INFO: deleting *v1.ClusterRole: external-attacher-runner-provisioning-4690 Nov 11 21:13:04.848: INFO: deleting *v1.ClusterRoleBinding: csi-attacher-role-provisioning-4690 Nov 11 21:13:04.912: INFO: deleting *v1.Role: provisioning-4690/external-attacher-cfg-provisioning-4690 Nov 11 21:13:05.054: INFO: deleting *v1.RoleBinding: provisioning-4690/csi-attacher-role-cfg Nov 11 21:13:05.121: INFO: deleting *v1.ServiceAccount: provisioning-4690/csi-provisioner Nov 11 21:13:05.191: INFO: deleting *v1.ClusterRole: external-provisioner-runner-provisioning-4690 Nov 11 21:13:05.266: INFO: deleting *v1.ClusterRoleBinding: csi-provisioner-role-provisioning-4690 Nov 11 21:13:05.334: INFO: deleting *v1.Role: provisioning-4690/external-provisioner-cfg-provisioning-4690 Nov 11 21:13:05.456: INFO: deleting *v1.RoleBinding: provisioning-4690/csi-provisioner-role-cfg Nov 11 21:13:05.524: INFO: deleting *v1.ServiceAccount: provisioning-4690/csi-snapshotter Nov 11 21:13:05.589: INFO: deleting *v1.ClusterRole: external-snapshotter-runner-provisioning-4690 Nov 11 21:13:05.653: INFO: deleting *v1.ClusterRoleBinding: csi-snapshotter-role-provisioning-4690 Nov 11 21:13:05.717: INFO: deleting *v1.Role: provisioning-4690/external-snapshotter-leaderelection-provisioning-4690 Nov 11 21:13:05.782: INFO: deleting *v1.RoleBinding: provisioning-4690/external-snapshotter-leaderelection Nov 11 21:13:05.855: INFO: deleting *v1.ServiceAccount: provisioning-4690/csi-resizer Nov 11 21:13:05.925: INFO: deleting *v1.ClusterRole: external-resizer-runner-provisioning-4690 Nov 11 21:13:05.988: INFO: deleting *v1.ClusterRoleBinding: csi-resizer-role-provisioning-4690 Nov 11 21:13:06.053: INFO: deleting *v1.Role: provisioning-4690/external-resizer-cfg-provisioning-4690 Nov 11 21:13:06.154: INFO: deleting *v1.RoleBinding: provisioning-4690/csi-resizer-role-cfg Nov 11 21:13:06.220: INFO: deleting *v1.Service: provisioning-4690/csi-hostpath-attacher Nov 11 21:13:06.325: INFO: deleting *v1.StatefulSet: provisioning-4690/csi-hostpath-attacher Nov 11 21:13:06.389: INFO: deleting *v1beta1.CSIDriver: csi-hostpath-provisioning-4690 Nov 11 21:13:06.461: INFO: deleting *v1.Service: provisioning-4690/csi-hostpathplugin Nov 11 21:13:06.574: INFO: deleting *v1.StatefulSet: provisioning-4690/csi-hostpathplugin Nov 11 21:13:06.639: INFO: deleting *v1.Service: provisioning-4690/csi-hostpath-provisioner Nov 11 21:13:06.734: INFO: deleting *v1.StatefulSet: provisioning-4690/csi-hostpath-provisioner Nov 11 21:13:06.798: INFO: deleting *v1.Service: provisioning-4690/csi-hostpath-resizer Nov 11 21:13:06.897: INFO: deleting *v1.StatefulSet: provisioning-4690/csi-hostpath-resizer Nov 11 21:13:06.965: INFO: deleting *v1.Service: provisioning-4690/csi-snapshotter Nov 11 21:13:07.127: INFO: deleting *v1.StatefulSet: provisioning-4690/csi-snapshotter Nov 11 21:13:07.194: INFO: deleting *v1.ClusterRoleBinding: psp-csi-hostpath-role-provisioning-4690 [AfterEach] [Testpattern: Dynamic PV (default fs)] subPath test/e2e/framework/framework.go:152 �[1mSTEP�[0m: Collecting events from namespace "provisioning-4690". �[1mSTEP�[0m: Found 79 events. Nov 11 21:13:07.329: INFO: At 2019-11-11 21:02:37 +0000 UTC - event for csi-hostpath-attacher: {statefulset-controller } SuccessfulCreate: create Pod csi-hostpath-attacher-0 in StatefulSet csi-hostpath-attacher successful Nov 11 21:13:07.329: INFO: At 2019-11-11 21:02:37 +0000 UTC - event for csi-hostpath-provisioner: {statefulset-controller } SuccessfulCreate: create Pod csi-hostpath-provisioner-0 in StatefulSet csi-hostpath-provisioner successful Nov 11 21:13:07.329: INFO: At 2019-11-11 21:02:37 +0000 UTC - event for csi-hostpath-resizer: {statefulset-controller } SuccessfulCreate: create Pod csi-hostpath-resizer-0 in StatefulSet csi-hostpath-resizer successful Nov 11 21:13:07.329: INFO: At 2019-11-11 21:02:37 +0000 UTC - event for csi-hostpathplugin: {statefulset-controller } SuccessfulCreate: create Pod csi-hostpathplugin-0 in StatefulSet csi-hostpathplugin successful Nov 11 21:13:07.329: INFO: At 2019-11-11 21:02:38 +0000 UTC - event for csi-hostpathhx7qn: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "csi-hostpath-provisioning-4690" or manually created by system administrator Nov 11 21:13:07.329: INFO: At 2019-11-11 21:02:38 +0000 UTC - event for csi-snapshotter: {statefulset-controller } SuccessfulCreate: create Pod csi-snapshotter-0 in StatefulSet csi-snapshotter successful Nov 11 21:13:07.329: INFO: At 2019-11-11 21:02:40 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:07.329: INFO: At 2019-11-11 21:02:44 +0000 UTC - event for csi-hostpathplugin: {statefulset-controller } FailedCreate: create Pod csi-hostpathplugin-0 in StatefulSet csi-hostpathplugin failed error: The POST operation against Pod could not be completed at this time, please try again. Nov 11 21:13:07.329: INFO: At 2019-11-11 21:02:44 +0000 UTC - event for csi-hostpathplugin: {statefulset-controller } SuccessfulDelete: delete Pod csi-hostpathplugin-0 in StatefulSet csi-hostpathplugin successful Nov 11 21:13:07.329: INFO: At 2019-11-11 21:02:44 +0000 UTC - event for csi-hostpathplugin: {statefulset-controller } RecreatingFailedPod: StatefulSet provisioning-4690/csi-hostpathplugin is recreating failed Pod csi-hostpathplugin-0 Nov 11 21:13:07.329: INFO: At 2019-11-11 21:02:46 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:07.329: INFO: At 2019-11-11 21:02:48 +0000 UTC - event for csi-hostpath-attacher-0: {kubelet k8s-agentpool-35271922-vmss000000} Pulling: Pulling image "quay.io/k8scsi/csi-attacher:v1.2.0" Nov 11 21:13:07.329: INFO: At 2019-11-11 21:02:50 +0000 UTC - event for csi-hostpath-attacher-0: {kubelet k8s-agentpool-35271922-vmss000000} Pulled: Successfully pulled image "quay.io/k8scsi/csi-attacher:v1.2.0" Nov 11 21:13:07.329: INFO: At 2019-11-11 21:02:50 +0000 UTC - event for csi-hostpath-provisioner-0: {kubelet k8s-agentpool-35271922-vmss000000} Pulling: Pulling image "quay.io/k8scsi/csi-provisioner:v1.4.0-rc1" Nov 11 21:13:07.329: INFO: At 2019-11-11 21:02:50 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:07.329: INFO: At 2019-11-11 21:02:55 +0000 UTC - event for csi-hostpath-resizer-0: {kubelet k8s-agentpool-35271922-vmss000000} Pulling: Pulling image "quay.io/k8scsi/csi-resizer:v0.2.0" Nov 11 21:13:07.329: INFO: At 2019-11-11 21:02:56 +0000 UTC - event for csi-hostpath-attacher-0: {kubelet k8s-agentpool-35271922-vmss000000} Created: Created container csi-attacher Nov 11 21:13:07.329: INFO: At 2019-11-11 21:02:57 +0000 UTC - event for csi-hostpath-provisioner-0: {kubelet k8s-agentpool-35271922-vmss000000} Pulled: Successfully pulled image "quay.io/k8scsi/csi-provisioner:v1.4.0-rc1" Nov 11 21:13:07.329: INFO: At 2019-11-11 21:03:01 +0000 UTC - event for csi-hostpath-attacher-0: {kubelet k8s-agentpool-35271922-vmss000000} Started: Started container csi-attacher Nov 11 21:13:07.329: INFO: At 2019-11-11 21:03:01 +0000 UTC - event for csi-hostpath-provisioner-0: {kubelet k8s-agentpool-35271922-vmss000000} Created: Created container csi-provisioner Nov 11 21:13:07.329: INFO: At 2019-11-11 21:03:01 +0000 UTC - event for csi-hostpath-resizer-0: {kubelet k8s-agentpool-35271922-vmss000000} Pulled: Successfully pulled image "quay.io/k8scsi/csi-resizer:v0.2.0" Nov 11 21:13:07.329: INFO: At 2019-11-11 21:03:01 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:07.329: INFO: At 2019-11-11 21:03:04 +0000 UTC - event for csi-snapshotter-0: {kubelet k8s-agentpool-35271922-vmss000000} Pulling: Pulling image "quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1" Nov 11 21:13:07.329: INFO: At 2019-11-11 21:03:06 +0000 UTC - event for csi-hostpath-provisioner-0: {kubelet k8s-agentpool-35271922-vmss000000} Started: Started container csi-provisioner Nov 11 21:13:07.329: INFO: At 2019-11-11 21:03:08 +0000 UTC - event for csi-hostpath-resizer-0: {kubelet k8s-agentpool-35271922-vmss000000} Created: Created container csi-resizer Nov 11 21:13:07.329: INFO: At 2019-11-11 21:03:10 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:07.329: INFO: At 2019-11-11 21:03:10 +0000 UTC - event for csi-snapshotter-0: {kubelet k8s-agentpool-35271922-vmss000000} Pulled: Successfully pulled image "quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1" Nov 11 21:13:07.329: INFO: At 2019-11-11 21:03:13 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:07.329: INFO: At 2019-11-11 21:03:16 +0000 UTC - event for csi-hostpath-resizer-0: {kubelet k8s-agentpool-35271922-vmss000000} Started: Started container csi-resizer Nov 11 21:13:07.329: INFO: At 2019-11-11 21:03:16 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:07.329: INFO: At 2019-11-11 21:03:16 +0000 UTC - event for csi-snapshotter-0: {kubelet k8s-agentpool-35271922-vmss000000} Created: Created container csi-snapshotter Nov 11 21:13:07.329: INFO: At 2019-11-11 21:03:18 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:07.329: INFO: At 2019-11-11 21:03:20 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:07.329: INFO: At 2019-11-11 21:03:22 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:07.329: INFO: At 2019-11-11 21:03:22 +0000 UTC - event for csi-snapshotter-0: {kubelet k8s-agentpool-35271922-vmss000000} Started: Started container csi-snapshotter Nov 11 21:13:07.329: INFO: At 2019-11-11 21:03:24 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:07.329: INFO: At 2019-11-11 21:03:25 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:07.329: INFO: At 2019-11-11 21:03:29 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:07.329: INFO: At 2019-11-11 21:03:31 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:07.329: INFO: At 2019-11-11 21:03:33 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:07.329: INFO: At 2019-11-11 21:03:36 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:07.329: INFO: At 2019-11-11 21:03:38 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:07.329: INFO: At 2019-11-11 21:03:39 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:07.329: INFO: At 2019-11-11 21:03:49 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:07.329: INFO: At 2019-11-11 21:03:55 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:07.329: INFO: At 2019-11-11 21:04:04 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:07.329: INFO: At 2019-11-11 21:04:10 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:07.329: INFO: At 2019-11-11 21:04:11 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:07.329: INFO: At 2019-11-11 21:04:11 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:07.329: INFO: At 2019-11-11 21:04:13 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:07.329: INFO: At 2019-11-11 21:04:16 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:07.329: INFO: At 2019-11-11 21:04:20 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:13:07.329: INFO: At 2019-11-11 21:05:08 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} Pulling: Pulling image "quay.io/k8scsi/csi-node-driver-registrar:v1.1.0" Nov 11 21:13:07.329: INFO: At 2019-11-11 21:05:09 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} Pulled: Successfully pulled image "quay.io/k8scsi/csi-node-driver-registrar:v1.1.0" Nov 11 21:13:07.329: INFO: At 2019-11-11 21:05:12 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} Created: Created container node-driver-registrar Nov 11 21:13:07.329: INFO: At 2019-11-11 21:05:13 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} Started: Started container node-driver-registrar Nov 11 21:13:07.329: INFO: At 2019-11-11 21:05:13 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} Pulling: Pulling image "quay.io/k8scsi/hostpathplugin:v1.2.0-rc5" Nov 11 21:13:07.329: INFO: At 2019-11-11 21:05:16 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} Pulled: Successfully pulled image "quay.io/k8scsi/hostpathplugin:v1.2.0-rc5" Nov 11 21:13:07.329: INFO: At 2019-11-11 21:05:18 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} Created: Created container hostpath Nov 11 21:13:07.329: INFO: At 2019-11-11 21:05:22 +0000 UTC - event for csi-hostpathhx7qn: {csi-hostpath-provisioning-4690_csi-hostpath-provisioner-0_729a44d8-30d1-49d5-9b44-af86c2e12f4e } Provisioning: External provisioner is provisioning volume for claim "provisioning-4690/csi-hostpathhx7qn" Nov 11 21:13:07.329: INFO: At 2019-11-11 21:05:22 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} Pulling: Pulling image "quay.io/k8scsi/livenessprobe:v1.1.0" Nov 11 21:13:07.329: INFO: At 2019-11-11 21:05:22 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} Started: Started container hostpath Nov 11 21:13:07.329: INFO: At 2019-11-11 21:05:23 +0000 UTC - event for pod-subpath-test-csi-hostpath-dynamicpv-ns5x: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-525180af-f06e-45d6-b287-04757cf4b323" Nov 11 21:13:07.329: INFO: At 2019-11-11 21:05:24 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} Pulled: Successfully pulled image "quay.io/k8scsi/livenessprobe:v1.1.0" Nov 11 21:13:07.329: INFO: At 2019-11-11 21:05:29 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} Created: Created container liveness-probe Nov 11 21:13:07.329: INFO: At 2019-11-11 21:05:33 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} Started: Started container liveness-probe Nov 11 21:13:07.329: INFO: At 2019-11-11 21:05:56 +0000 UTC - event for pod-subpath-test-csi-hostpath-dynamicpv-ns5x: {kubelet k8s-agentpool-35271922-vmss000000} Pulling: Pulling image "docker.io/library/busybox:1.29" Nov 11 21:13:07.329: INFO: At 2019-11-11 21:05:57 +0000 UTC - event for pod-subpath-test-csi-hostpath-dynamicpv-ns5x: {kubelet k8s-agentpool-35271922-vmss000000} Pulled: Successfully pulled image "docker.io/library/busybox:1.29" Nov 11 21:13:07.329: INFO: At 2019-11-11 21:06:01 +0000 UTC - event for pod-subpath-test-csi-hostpath-dynamicpv-ns5x: {kubelet k8s-agentpool-35271922-vmss000000} Created: Created container init-volume-csi-hostpath-dynamicpv-ns5x Nov 11 21:13:07.329: INFO: At 2019-11-11 21:06:23 +0000 UTC - event for pod-subpath-test-csi-hostpath-dynamicpv-ns5x: {kubelet k8s-agentpool-35271922-vmss000000} Started: Started container init-volume-csi-hostpath-dynamicpv-ns5x Nov 11 21:13:07.329: INFO: At 2019-11-11 21:06:34 +0000 UTC - event for pod-subpath-test-csi-hostpath-dynamicpv-ns5x: {kubelet k8s-agentpool-35271922-vmss000000} Pulling: Pulling image "gcr.io/kubernetes-e2e-test-images/mounttest:1.0" Nov 11 21:13:07.329: INFO: At 2019-11-11 21:06:40 +0000 UTC - event for pod-subpath-test-csi-hostpath-dynamicpv-ns5x: {kubelet k8s-agentpool-35271922-vmss000000} Pulled: Successfully pulled image "gcr.io/kubernetes-e2e-test-images/mounttest:1.0" Nov 11 21:13:07.329: INFO: At 2019-11-11 21:06:45 +0000 UTC - event for pod-subpath-test-csi-hostpath-dynamicpv-ns5x: {kubelet k8s-agentpool-35271922-vmss000000} Created: Created container test-init-subpath-csi-hostpath-dynamicpv-ns5x Nov 11 21:13:07.329: INFO: At 2019-11-11 21:06:54 +0000 UTC - event for pod-subpath-test-csi-hostpath-dynamicpv-ns5x: {kubelet k8s-agentpool-35271922-vmss000000} Started: Started container test-init-subpath-csi-hostpath-dynamicpv-ns5x Nov 11 21:13:07.329: INFO: At 2019-11-11 21:06:55 +0000 UTC - event for pod-subpath-test-csi-hostpath-dynamicpv-ns5x: {kubelet k8s-agentpool-35271922-vmss000000} Pulling: Pulling image "gcr.io/kubernetes-e2e-test-images/mounttest:1.0" Nov 11 21:13:07.329: INFO: At 2019-11-11 21:06:56 +0000 UTC - event for pod-subpath-test-csi-hostpath-dynamicpv-ns5x: {kubelet k8s-agentpool-35271922-vmss000000} Pulled: Successfully pulled image "gcr.io/kubernetes-e2e-test-images/mounttest:1.0" Nov 11 21:13:07.329: INFO: At 2019-11-11 21:07:02 +0000 UTC - event for pod-subpath-test-csi-hostpath-dynamicpv-ns5x: {kubelet k8s-agentpool-35271922-vmss000000} Created: Created container test-container-subpath-csi-hostpath-dynamicpv-ns5x Nov 11 21:13:07.329: INFO: At 2019-11-11 21:07:09 +0000 UTC - event for pod-subpath-test-csi-hostpath-dynamicpv-ns5x: {kubelet k8s-agentpool-35271922-vmss000000} Started: Started container test-container-subpath-csi-hostpath-dynamicpv-ns5x Nov 11 21:13:07.329: INFO: At 2019-11-11 21:09:57 +0000 UTC - event for pod-subpath-test-csi-hostpath-dynamicpv-ns5x: {kubelet k8s-agentpool-35271922-vmss000000} FailedMount: Unable to attach or mount volumes: unmounted volumes=[test-volume], unattached volumes=[test-volume liveness-probe-volume default-token-crvm4]: timed out waiting for the condition Nov 11 21:13:07.388: INFO: POD NODE PHASE GRACE CONDITIONS Nov 11 21:13:07.388: INFO: csi-hostpath-attacher-0 k8s-agentpool-35271922-vmss000000 Running 30s [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:02:37 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:03:35 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:03:35 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:02:37 +0000 UTC }] Nov 11 21:13:07.388: INFO: csi-hostpath-provisioner-0 k8s-agentpool-35271922-vmss000000 Running 30s [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:02:40 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:03:26 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:03:26 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:02:40 +0000 UTC }] Nov 11 21:13:07.388: INFO: csi-hostpath-resizer-0 k8s-agentpool-35271922-vmss000000 Running 30s [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:02:40 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:03:42 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:03:42 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:02:40 +0000 UTC }] Nov 11 21:13:07.388: INFO: csi-hostpathplugin-0 k8s-agentpool-35271922-vmss000000 Running 30s [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:04:26 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:05:34 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:05:34 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:04:26 +0000 UTC }] Nov 11 21:13:07.388: INFO: csi-snapshotter-0 k8s-agentpool-35271922-vmss000000 Running 30s [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:02:41 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:03:35 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:03:35 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:02:41 +0000 UTC }] Nov 11 21:13:07.388: INFO: Nov 11 21:13:07.503: INFO: Logging node info for node k8s-agentpool-35271922-vmss000000 Nov 11 21:13:07.560: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool-35271922-vmss000000 /api/v1/nodes/k8s-agentpool-35271922-vmss000000 04da751e-f23a-470e-b29d-e591113d7271 41796 0 2019-11-11 20:48:07 +0000 UTC <nil> <nil> map[agentpool:agentpool beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubelet_cleanup:true kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool-35271922-vmss000000 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: storageprofile:managed storagetier:Premium_LRS] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-2030":"k8s-agentpool-35271922-vmss000000","csi-hostpath-ephemeral-7436":"k8s-agentpool-35271922-vmss000000","csi-hostpath-provisioning-3597":"k8s-agentpool-35271922-vmss000000","csi-hostpath-provisioning-9334":"k8s-agentpool-35271922-vmss000000","csi-hostpath-volume-5092":"k8s-agentpool-35271922-vmss000000","csi-hostpath-volume-expand-1449":"k8s-agentpool-35271922-vmss000000","csi-mock-csi-mock-volumes-1829":"csi-mock-csi-mock-volumes-1829","csi-mock-csi-mock-volumes-2481":"csi-mock-csi-mock-volumes-2481","csi-mock-csi-mock-volumes-763":"csi-mock-csi-mock-volumes-763","csi-mock-csi-mock-volumes-9721":"csi-mock-csi-mock-volumes-9721"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-35271922-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16797044736 0} {<nil>} 16403364Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16010612736 0} {<nil>} 15635364Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:13:03 +0000 UTC,LastTransitionTime:2019-11-11 20:47:58 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:13:03 +0000 UTC,LastTransitionTime:2019-11-11 20:47:58 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:13:03 +0000 UTC,LastTransitionTime:2019-11-11 20:47:58 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:13:03 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.248.0.4,},NodeAddress{Type:Hostname,Address:k8s-agentpool-35271922-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:84c06f3ee41a47aaa558c81f34711895,SystemUUID:9BB24974-637B-EE4C-A221-0F74BB743424,BootID:59f7eb0c-c09c-4eae-a14b-0710f9e14004,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[quay.io/kubernetes_incubator/nfs-provisioner@sha256:df762117e3c891f2d2ddff46ecb0776ba1f9f3c44cfd7739b0683bcd7a7954a8 quay.io/kubernetes_incubator/nfs-provisioner:v2.2.2],SizeBytes:391772778,},ContainerImage{Names:[gluster/glusterdynamic-provisioner@sha256:90067cb05a7d217651e84576935934fd9dcff8e6bcdcbaa416bbf36fcd09dbd1 gluster/glusterdynamic-provisioner:v1.0],SizeBytes:373281573,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/volume/nfs@sha256:c2ad734346f608a5f7d69cfded93c4e8094069320657bd372d12ba21dea3ea71 gcr.io/kubernetes-e2e-test-images/volume/nfs:1.0],SizeBytes:225358913,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/jessie-dnsutils@sha256:ad583e33cb284f7ef046673809b146ec4053cda19b54a85d2b180a86169715eb gcr.io/kubernetes-e2e-test-images/jessie-dnsutils:1.0],SizeBytes:195659796,},ContainerImage{Names:[httpd@sha256:addd70e4ee83f3bc9a4c1c7c41e37927ba47faf639312fc936df3afad7926f5a httpd:2.4.39-alpine],SizeBytes:126894770,},ContainerImage{Names:[httpd@sha256:eb8ccf084cf3e80eece1add239effefd171eb39adbc154d33c14260d905d4060 httpd:2.4.38-alpine],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1],SizeBytes:121711221,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727 gcr.io/kubernetes-e2e-test-images/agnhost:2.6],SizeBytes:57345321,},ContainerImage{Names:[quay.io/k8scsi/csi-provisioner@sha256:0efcb424f1dde9b9fb11a1a14f2e48ab47e1c3f08bc3a929990dcfcb1f7ab34f quay.io/k8scsi/csi-provisioner:v1.4.0-rc1],SizeBytes:54431016,},ContainerImage{Names:[quay.io/k8scsi/csi-snapshotter@sha256:e3d3e742e32d00488fdb401045b9b1d033d7ca0ab6e760f77b24750fc95e5f70 quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1],SizeBytes:51703561,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0],SizeBytes:46226754,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:0aba670b4d9d6b2e720bbf575d733156c676b693ca26501235444490300db838 quay.io/k8scsi/csi-attacher:v1.1.0],SizeBytes:42839085,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:7d46fb6eb8b890dc546029d1565d502b4a1d974d33625c6ee2bc7991b77fc1a1 quay.io/k8scsi/csi-resizer:v0.2.0],SizeBytes:42817100,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:f315c9042e56def3c05c6b04fe79ec9da6d39ddc557ca365a76cf35964ea08b6 quay.io/k8scsi/csi-resizer:v0.1.0],SizeBytes:42623056,},ContainerImage{Names:[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0],SizeBytes:42488424,},ContainerImage{Names:[quay.io/k8scsi/hostpathplugin@sha256:b4826e492fc1762fceaf9726f41575ca0a4567864d3d235da874818de18039de quay.io/k8scsi/hostpathplugin:v1.2.0-rc5],SizeBytes:28761497,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7],SizeBytes:22909487,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/echoserver@sha256:e9ba514b896cdf559eef8788b66c2c3ee55f3572df617647b4b0d8b6bf81cf19 gcr.io/kubernetes-e2e-test-images/echoserver:2.2],SizeBytes:21692741,},ContainerImage{Names:[quay.io/k8scsi/mock-driver@sha256:e0eed916b7d970bad2b7d9875f9ad16932f987f0f3d91ec5d86da68b0b5cc9d1 quay.io/k8scsi/mock-driver:v2.1.0],SizeBytes:16226335,},ContainerImage{Names:[nginx@sha256:485b610fefec7ff6c463ced9623314a04ed67e3945b9c08d7e53a47f6d108dc7 nginx:1.14-alpine],SizeBytes:16032814,},ContainerImage{Names:[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0],SizeBytes:15815995,},ContainerImage{Names:[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0],SizeBytes:14967303,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/dnsutils@sha256:2abeee84efb79c14d731966e034af33bf324d3b26ca28497555511ff094b3ddd gcr.io/kubernetes-e2e-test-images/dnsutils:1.1],SizeBytes:9349974,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nautilus@sha256:33a732d4c42a266912a5091598a0f07653c9134db4b8d571690d8afd509e0bfc gcr.io/kubernetes-e2e-test-images/nautilus:1.0],SizeBytes:4753501,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/test-webserver@sha256:7f93d6e32798ff28bc6289254d0c2867fe2c849c8e46edc50f8624734309812e gcr.io/kubernetes-e2e-test-images/test-webserver:1.0],SizeBytes:4732240,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 gcr.io/kubernetes-e2e-test-images/mounttest:1.0],SizeBytes:1563521,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29],SizeBytes:1154361,},ContainerImage{Names:[busybox@sha256:bbc3a03235220b170ba48a157dd097dd1379299370e1ed99ce976df0355d24f0 busybox:1.27],SizeBytes:1129289,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea k8s.gcr.io/pause-amd64:3.1 k8s.gcr.io/pause:3.1],SizeBytes:742472,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-763^4],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-763^4,DevicePath:,},},Config:nil,},} Nov 11 21:13:07.561: INFO: Logging kubelet events for node k8s-agentpool-35271922-vmss000000 Nov 11 21:13:07.622: INFO: Logging pods the kubelet thinks is on node k8s-agentpool-35271922-vmss000000 Nov 11 21:13:07.749: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-sqdt4 started at 2019-11-11 21:09:54 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.749: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: true, restart count 0 Nov 11 21:13:07.749: INFO: csi-snapshotter-0 started at 2019-11-11 21:10:29 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.749: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 11 21:13:07.749: INFO: csi-hostpathplugin-0 started at 2019-11-11 21:04:26 +0000 UTC (0+3 container statuses recorded) Nov 11 21:13:07.749: INFO: Container hostpath ready: true, restart count 0 Nov 11 21:13:07.749: INFO: Container liveness-probe ready: true, restart count 0 Nov 11 21:13:07.749: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 11 21:13:07.749: INFO: externalname-service-2mr2n started at 2019-11-11 21:10:33 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.749: INFO: Container externalname-service ready: true, restart count 0 Nov 11 21:13:07.749: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:07:21 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.749: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:13:07.749: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:11:47 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.749: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:13:07.749: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-7ndbp started at 2019-11-11 21:09:58 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.749: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: true, restart count 0 Nov 11 21:13:07.749: INFO: webserver-dc8bd894f-vm4pq started at 2019-11-11 21:12:41 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.749: INFO: Container httpd ready: false, restart count 0 Nov 11 21:13:07.749: INFO: csi-hostpathplugin-0 started at <nil> (0+0 container statuses recorded) Nov 11 21:13:07.749: INFO: pod-subpath-test-local-preprovisionedpv-qjr5 started at 2019-11-11 21:10:07 +0000 UTC (2+1 container statuses recorded) Nov 11 21:13:07.749: INFO: Init container init-volume-local-preprovisionedpv-qjr5 ready: true, restart count 0 Nov 11 21:13:07.749: INFO: Init container test-init-volume-local-preprovisionedpv-qjr5 ready: true, restart count 0 Nov 11 21:13:07.749: INFO: Container test-container-subpath-local-preprovisionedpv-qjr5 ready: false, restart count 0 Nov 11 21:13:07.749: INFO: csi-hostpath-resizer-0 started at 2019-11-11 21:10:28 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.749: INFO: Container csi-resizer ready: true, restart count 0 Nov 11 21:13:07.749: INFO: execpods8j8z started at 2019-11-11 21:11:52 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.749: INFO: Container agnhost-pause ready: true, restart count 0 Nov 11 21:13:07.749: INFO: pod-subpath-test-local-preprovisionedpv-rqnf started at 2019-11-11 21:11:54 +0000 UTC (0+2 container statuses recorded) Nov 11 21:13:07.749: INFO: Container test-container-subpath-local-preprovisionedpv-rqnf ready: false, restart count 0 Nov 11 21:13:07.749: INFO: Container test-container-volume-local-preprovisionedpv-rqnf ready: true, restart count 0 Nov 11 21:13:07.749: INFO: csi-hostpath-resizer-0 started at 2019-11-11 21:07:52 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.749: INFO: Container csi-resizer ready: true, restart count 0 Nov 11 21:13:07.749: INFO: external-provisioner-6rv66 started at 2019-11-11 21:11:45 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.749: INFO: Container nfs-provisioner ready: true, restart count 0 Nov 11 21:13:07.749: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-n58zl started at 2019-11-11 21:09:53 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.749: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: true, restart count 0 Nov 11 21:13:07.749: INFO: blobfuse-flexvol-installer-dmnhw started at 2019-11-11 20:48:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.749: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 Nov 11 21:13:07.749: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-vdczm started at 2019-11-11 21:10:00 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.749: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: true, restart count 0 Nov 11 21:13:07.749: INFO: csi-hostpath-provisioner-0 started at 2019-11-11 21:02:40 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.749: INFO: Container csi-provisioner ready: true, restart count 0 Nov 11 21:13:07.749: INFO: csi-hostpath-resizer-0 started at 2019-11-11 21:08:37 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.749: INFO: Container csi-resizer ready: true, restart count 0 Nov 11 21:13:07.749: INFO: coredns-87f5d796-6mz2l started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.749: INFO: Container coredns ready: true, restart count 0 Nov 11 21:13:07.749: INFO: csi-hostpath-attacher-0 started at 2019-11-11 21:02:37 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.749: INFO: Container csi-attacher ready: true, restart count 0 Nov 11 21:13:07.749: INFO: security-context-50ac59ee-3233-4a07-9d18-a85cc2c4304d started at 2019-11-11 21:12:34 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.749: INFO: Container write-pod ready: false, restart count 0 Nov 11 21:13:07.749: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-695c7 started at 2019-11-11 21:09:56 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.749: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: true, restart count 0 Nov 11 21:13:07.749: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:11:30 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.749: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:13:07.749: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-77l5m started at 2019-11-11 21:09:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.749: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: true, restart count 0 Nov 11 21:13:07.749: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-fmq2z started at 2019-11-11 21:09:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.749: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: true, restart count 0 Nov 11 21:13:07.749: INFO: webserver-77c9474f75-ff62s started at 2019-11-11 21:12:41 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.749: INFO: Container httpd ready: false, restart count 0 Nov 11 21:13:07.749: INFO: csi-hostpath-provisioner-0 started at 2019-11-11 21:07:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.750: INFO: Container csi-provisioner ready: true, restart count 0 Nov 11 21:13:07.750: INFO: csi-hostpath-attacher-0 started at 2019-11-11 21:08:34 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.750: INFO: Container csi-attacher ready: true, restart count 0 Nov 11 21:13:07.750: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-6wrg6 started at 2019-11-11 21:09:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.750: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: true, restart count 0 Nov 11 21:13:07.750: INFO: local-client started at 2019-11-11 21:10:53 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.750: INFO: Container local-client ready: true, restart count 0 Nov 11 21:13:07.750: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:11:48 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.750: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:13:07.750: INFO: kube-proxy-nwdpz started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.750: INFO: Container kube-proxy ready: true, restart count 0 Nov 11 21:13:07.750: INFO: csi-snapshotter-0 started at 2019-11-11 21:07:53 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.750: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 11 21:13:07.750: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-h7bv7 started at 2019-11-11 21:10:01 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.750: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: true, restart count 0 Nov 11 21:13:07.750: INFO: csi-mockplugin-attacher-0 started at 2019-11-11 21:11:13 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.750: INFO: Container csi-attacher ready: true, restart count 0 Nov 11 21:13:07.750: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:10:12 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.750: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:13:07.750: INFO: pvc-volume-tester-894sb started at 2019-11-11 21:12:38 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.750: INFO: Container volume-tester ready: false, restart count 0 Nov 11 21:13:07.750: INFO: azure-ip-masq-agent-c4mb2 started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.750: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:13:07.750: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-rlsfg started at 2019-11-11 21:09:55 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.750: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: true, restart count 0 Nov 11 21:13:07.750: INFO: sysctl-41c353e7-2538-4d6a-88be-d351690575c5 started at 2019-11-11 21:11:38 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.750: INFO: Container test-container ready: false, restart count 0 Nov 11 21:13:07.750: INFO: csi-hostpath-provisioner-0 started at 2019-11-11 21:10:26 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.750: INFO: Container csi-provisioner ready: true, restart count 0 Nov 11 21:13:07.750: INFO: ss2-0 started at 2019-11-11 21:10:44 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.750: INFO: Container webserver ready: true, restart count 0 Nov 11 21:13:07.750: INFO: csi-mockplugin-0 started at 2019-11-11 21:11:12 +0000 UTC (0+3 container statuses recorded) Nov 11 21:13:07.750: INFO: Container csi-provisioner ready: true, restart count 0 Nov 11 21:13:07.750: INFO: Container driver-registrar ready: true, restart count 0 Nov 11 21:13:07.750: INFO: Container mock ready: true, restart count 0 Nov 11 21:13:07.750: INFO: csi-mockplugin-resizer-0 started at 2019-11-11 21:11:14 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.750: INFO: Container csi-resizer ready: true, restart count 0 Nov 11 21:13:07.750: INFO: csi-hostpath-provisioner-0 started at 2019-11-11 21:08:36 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.750: INFO: Container csi-provisioner ready: true, restart count 0 Nov 11 21:13:07.750: INFO: csi-hostpathplugin-0 started at 2019-11-11 21:13:06 +0000 UTC (0+0 container statuses recorded) Nov 11 21:13:07.750: INFO: csi-snapshotter-0 started at 2019-11-11 21:02:41 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.750: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 11 21:13:07.750: INFO: keyvault-flexvolume-zzp7p started at 2019-11-11 20:48:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.750: INFO: Container keyvault-flexvolume ready: true, restart count 0 Nov 11 21:13:07.750: INFO: csi-hostpath-attacher-0 started at 2019-11-11 21:07:46 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.750: INFO: Container csi-attacher ready: true, restart count 0 Nov 11 21:13:07.750: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:11:34 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.750: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:13:07.750: INFO: csi-hostpath-resizer-0 started at 2019-11-11 21:02:40 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.750: INFO: Container csi-resizer ready: true, restart count 0 Nov 11 21:13:07.750: INFO: csi-hostpath-attacher-0 started at 2019-11-11 21:10:24 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.750: INFO: Container csi-attacher ready: true, restart count 0 Nov 11 21:13:07.750: INFO: rs-cnbnm started at 2019-11-11 21:10:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.750: INFO: Container busybox ready: true, restart count 0 Nov 11 21:13:07.750: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:12:23 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.750: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:13:07.750: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:09:14 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.750: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:13:07.750: INFO: csi-snapshotter-0 started at 2019-11-11 21:08:38 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.750: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 11 21:13:07.750: INFO: kubernetes-dashboard-65966766b9-lv2x8 started at 2019-11-11 20:48:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.750: INFO: Container kubernetes-dashboard ready: true, restart count 0 Nov 11 21:13:07.750: INFO: update-demo-nautilus-qr298 started at 2019-11-11 21:07:18 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:07.750: INFO: Container update-demo ready: false, restart count 0 W1111 21:13:07.809187 93370 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:13:08.178: INFO: Latency metrics for node k8s-agentpool-35271922-vmss000000 Nov 11 21:13:08.178: INFO: Logging node info for node k8s-agentpool-35271922-vmss000001 Nov 11 21:13:08.234: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool-35271922-vmss000001 /api/v1/nodes/k8s-agentpool-35271922-vmss000001 af5ba0b6-4f7f-4468-9d3c-b3e116ca81ca 41314 0 2019-11-11 20:48:13 +0000 UTC <nil> <nil> map[agentpool:agentpool beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubelet_cleanup:true kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool-35271922-vmss000001 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: storageprofile:managed storagetier:Premium_LRS] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-8406":"k8s-agentpool-35271922-vmss000001","csi-hostpath-provisioning-2466":"k8s-agentpool-35271922-vmss000001","csi-hostpath-provisioning-265":"k8s-agentpool-35271922-vmss000001","csi-hostpath-provisioning-310":"k8s-agentpool-35271922-vmss000001","csi-hostpath-volume-expand-3983":"k8s-agentpool-35271922-vmss000001","csi-mock-csi-mock-volumes-2781":"csi-mock-csi-mock-volumes-2781","csi-mock-csi-mock-volumes-2838":"csi-mock-csi-mock-volumes-2838","csi-mock-csi-mock-volumes-3306":"csi-mock-csi-mock-volumes-3306","csi-mock-csi-mock-volumes-6769":"csi-mock-csi-mock-volumes-6769","csi-mock-csi-mock-volumes-694":"csi-mock-csi-mock-volumes-694","csi-mock-csi-mock-volumes-797":"csi-mock-csi-mock-volumes-797"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-35271922-vmss/virtualMachines/1,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16797044736 0} {<nil>} 16403364Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16010612736 0} {<nil>} 15635364Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:12:38 +0000 UTC,LastTransitionTime:2019-11-11 20:48:13 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:12:38 +0000 UTC,LastTransitionTime:2019-11-11 20:48:13 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:12:38 +0000 UTC,LastTransitionTime:2019-11-11 20:48:13 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:12:38 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.248.0.5,},NodeAddress{Type:Hostname,Address:k8s-agentpool-35271922-vmss000001,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:1df8c22200b84e64b76150d232ecb929,SystemUUID:B03EE801-7A2D-7140-BE20-B2C844575A73,BootID:d32206f7-48b0-46c5-a542-39721566b456,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[quay.io/kubernetes_incubator/nfs-provisioner@sha256:df762117e3c891f2d2ddff46ecb0776ba1f9f3c44cfd7739b0683bcd7a7954a8 quay.io/kubernetes_incubator/nfs-provisioner:v2.2.2],SizeBytes:391772778,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/volume/nfs@sha256:c2ad734346f608a5f7d69cfded93c4e8094069320657bd372d12ba21dea3ea71 gcr.io/kubernetes-e2e-test-images/volume/nfs:1.0],SizeBytes:225358913,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/jessie-dnsutils@sha256:ad583e33cb284f7ef046673809b146ec4053cda19b54a85d2b180a86169715eb gcr.io/kubernetes-e2e-test-images/jessie-dnsutils:1.0],SizeBytes:195659796,},ContainerImage{Names:[httpd@sha256:addd70e4ee83f3bc9a4c1c7c41e37927ba47faf639312fc936df3afad7926f5a httpd:2.4.39-alpine],SizeBytes:126894770,},ContainerImage{Names:[httpd@sha256:eb8ccf084cf3e80eece1add239effefd171eb39adbc154d33c14260d905d4060 httpd:2.4.38-alpine],SizeBytes:123781643,},ContainerImage{Names:[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0],SizeBytes:71821984,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727 gcr.io/kubernetes-e2e-test-images/agnhost:2.6],SizeBytes:57345321,},ContainerImage{Names:[quay.io/k8scsi/csi-provisioner@sha256:0efcb424f1dde9b9fb11a1a14f2e48ab47e1c3f08bc3a929990dcfcb1f7ab34f quay.io/k8scsi/csi-provisioner:v1.4.0-rc1],SizeBytes:54431016,},ContainerImage{Names:[quay.io/k8scsi/csi-snapshotter@sha256:e3d3e742e32d00488fdb401045b9b1d033d7ca0ab6e760f77b24750fc95e5f70 quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1],SizeBytes:51703561,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0],SizeBytes:46226754,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:0aba670b4d9d6b2e720bbf575d733156c676b693ca26501235444490300db838 quay.io/k8scsi/csi-attacher:v1.1.0],SizeBytes:42839085,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:7d46fb6eb8b890dc546029d1565d502b4a1d974d33625c6ee2bc7991b77fc1a1 quay.io/k8scsi/csi-resizer:v0.2.0],SizeBytes:42817100,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:f315c9042e56def3c05c6b04fe79ec9da6d39ddc557ca365a76cf35964ea08b6 quay.io/k8scsi/csi-resizer:v0.1.0],SizeBytes:42623056,},ContainerImage{Names:[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1],SizeBytes:42541759,},ContainerImage{Names:[gcr.io/google-containers/debian-base@sha256:6966a0aedd7592c18ff2dd803c08bd85780ee19f5e3a2e7cf908a4cd837afcde gcr.io/google-containers/debian-base:0.4.1],SizeBytes:42323657,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nonroot@sha256:d4ede5c74517090b6686219059118ed178cf4620f5db8781b32f806bb1e7395b gcr.io/kubernetes-e2e-test-images/nonroot:1.0],SizeBytes:42321438,},ContainerImage{Names:[redis@sha256:50899ea1ceed33fa03232f3ac57578a424faa1742c1ac9c7a7bdb95cdf19b858 redis:5.0.5-alpine],SizeBytes:29331594,},ContainerImage{Names:[quay.io/k8scsi/hostpathplugin@sha256:b4826e492fc1762fceaf9726f41575ca0a4567864d3d235da874818de18039de quay.io/k8scsi/hostpathplugin:v1.2.0-rc5],SizeBytes:28761497,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7],SizeBytes:22909487,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/echoserver@sha256:e9ba514b896cdf559eef8788b66c2c3ee55f3572df617647b4b0d8b6bf81cf19 gcr.io/kubernetes-e2e-test-images/echoserver:2.2],SizeBytes:21692741,},ContainerImage{Names:[quay.io/k8scsi/mock-driver@sha256:e0eed916b7d970bad2b7d9875f9ad16932f987f0f3d91ec5d86da68b0b5cc9d1 quay.io/k8scsi/mock-driver:v2.1.0],SizeBytes:16226335,},ContainerImage{Names:[nginx@sha256:485b610fefec7ff6c463ced9623314a04ed67e3945b9c08d7e53a47f6d108dc7 nginx:1.14-alpine],SizeBytes:16032814,},ContainerImage{Names:[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0],SizeBytes:15815995,},ContainerImage{Names:[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0],SizeBytes:14967303,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/dnsutils@sha256:2abeee84efb79c14d731966e034af33bf324d3b26ca28497555511ff094b3ddd gcr.io/kubernetes-e2e-test-images/dnsutils:1.1],SizeBytes:9349974,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nonewprivs@sha256:10066e9039219449fe3c81f38fe01928f87914150768ab81b62a468e51fa7411 gcr.io/kubernetes-e2e-test-images/nonewprivs:1.0],SizeBytes:6757579,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nautilus@sha256:33a732d4c42a266912a5091598a0f07653c9134db4b8d571690d8afd509e0bfc gcr.io/kubernetes-e2e-test-images/nautilus:1.0],SizeBytes:4753501,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/test-webserver@sha256:7f93d6e32798ff28bc6289254d0c2867fe2c849c8e46edc50f8624734309812e gcr.io/kubernetes-e2e-test-images/test-webserver:1.0],SizeBytes:4732240,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:4206620,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 gcr.io/kubernetes-e2e-test-images/mounttest:1.0],SizeBytes:1563521,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest-user@sha256:17319ca525ee003681fccf7e8c6b1b910ff4f49b653d939ac7f9b6e7c463933d gcr.io/kubernetes-e2e-test-images/mounttest-user:1.0],SizeBytes:1450451,},ContainerImage{Names:[busybox@sha256:1303dbf110c57f3edf68d9f5a16c082ec06c4cf7604831669faf2c712260b5a0 busybox:latest],SizeBytes:1219790,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29],SizeBytes:1154361,},ContainerImage{Names:[busybox@sha256:bbc3a03235220b170ba48a157dd097dd1379299370e1ed99ce976df0355d24f0 busybox:1.27],SizeBytes:1129289,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea k8s.gcr.io/pause-amd64:3.1 k8s.gcr.io/pause:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:13:08.235: INFO: Logging kubelet events for node k8s-agentpool-35271922-vmss000001 Nov 11 21:13:08.299: INFO: Logging pods the kubelet thinks is on node k8s-agentpool-35271922-vmss000001 Nov 11 21:13:08.484: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-xvlpv started at 2019-11-11 21:09:55 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:08.484: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: true, restart count 0 Nov 11 21:13:08.484: INFO: csi-mockplugin-0 started at 2019-11-11 21:07:06 +0000 UTC (0+3 container statuses recorded) Nov 11 21:13:08.484: INFO: Container csi-provisioner ready: true, restart count 0 Nov 11 21:13:08.484: INFO: Container driver-registrar ready: true, restart count 0 Nov 11 21:13:08.484: INFO: Container mock ready: true, restart count 0 Nov 11 21:13:08.484: INFO: azure-ip-masq-agent-9427k started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:08.484: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:13:08.484: INFO: hostexec-k8s-agentpool-35271922-vmss000001 started at 2019-11-11 21:12:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:08.484: INFO: Container agnhost ready: false, restart count 0 Nov 11 21:13:08.484: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-5xrt6 started at 2019-11-11 21:09:49 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:08.484: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: true, restart count 0 Nov 11 21:13:08.484: INFO: webserver-77c9474f75-b66lx started at 2019-11-11 21:12:43 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:08.484: INFO: Container httpd ready: false, restart count 0 Nov 11 21:13:08.484: INFO: var-expansion-cd36be16-75e0-4d5c-a5fc-6f731ce949bb started at 2019-11-11 21:12:55 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:08.484: INFO: Container dapi-container ready: false, restart count 0 Nov 11 21:13:08.484: INFO: blobfuse-flexvol-installer-tvkwl started at 2019-11-11 20:48:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:08.484: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 Nov 11 21:13:08.484: INFO: tiller-deploy-7559b6b885-jsnn2 started at 2019-11-11 20:48:53 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:08.484: INFO: Container tiller ready: true, restart count 0 Nov 11 21:13:08.484: INFO: webserver-dc8bd894f-5pfhd started at 2019-11-11 21:12:43 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:08.484: INFO: Container httpd ready: false, restart count 0 Nov 11 21:13:08.484: INFO: ss2-1 started at 2019-11-11 21:08:52 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:08.484: INFO: Container webserver ready: true, restart count 0 Nov 11 21:13:08.484: INFO: metrics-server-58ff8c5ddf-thxvp started at 2019-11-11 20:48:53 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:08.484: INFO: Container metrics-server ready: true, restart count 0 Nov 11 21:13:08.484: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-vzxn8 started at 2019-11-11 21:09:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:08.484: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: true, restart count 0 Nov 11 21:13:08.484: INFO: webserver-77c9474f75-zws9d started at 2019-11-11 21:12:37 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:08.484: INFO: Container httpd ready: false, restart count 0 Nov 11 21:13:08.484: INFO: externalname-service-nbf4b started at 2019-11-11 21:10:25 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:08.484: INFO: Container externalname-service ready: true, restart count 0 Nov 11 21:13:08.484: INFO: replace-1573506660-vrzrd started at 2019-11-11 21:11:09 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:08.484: INFO: Container c ready: true, restart count 0 Nov 11 21:13:08.484: INFO: csi-mockplugin-attacher-0 started at 2019-11-11 21:07:07 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:08.484: INFO: Container csi-attacher ready: false, restart count 0 Nov 11 21:13:08.484: INFO: external-provisioner-64xss started at 2019-11-11 21:12:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:08.484: INFO: Container nfs-provisioner ready: false, restart count 0 Nov 11 21:13:08.484: INFO: pod-prestop-hook-b9c29d46-a5ed-430d-beb3-947a38845e94 started at 2019-11-11 21:12:11 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:08.484: INFO: Container nginx ready: false, restart count 0 Nov 11 21:13:08.484: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-cbdrk started at 2019-11-11 21:09:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:08.484: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: true, restart count 0 Nov 11 21:13:08.484: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-m44k9 started at 2019-11-11 21:09:58 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:08.484: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: true, restart count 0 Nov 11 21:13:08.484: INFO: pod-ready started at 2019-11-11 21:10:19 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:08.484: INFO: Container pod-readiness-gate ready: true, restart count 0 Nov 11 21:13:08.484: INFO: webserver-dc8bd894f-ldzf4 started at 2019-11-11 21:12:36 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:08.484: INFO: Container httpd ready: false, restart count 0 Nov 11 21:13:08.484: INFO: webserver-6f55b98-6gv2w started at 2019-11-11 21:12:25 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:08.484: INFO: Container httpd ready: false, restart count 0 Nov 11 21:13:08.484: INFO: rs-k269h started at 2019-11-11 21:10:53 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:08.484: INFO: Container busybox ready: true, restart count 0 Nov 11 21:13:08.484: INFO: keyvault-flexvolume-2s2wq started at 2019-11-11 20:48:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:08.484: INFO: Container keyvault-flexvolume ready: true, restart count 0 Nov 11 21:13:08.484: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-sn584 started at 2019-11-11 21:09:53 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:08.484: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: false, restart count 0 Nov 11 21:13:08.484: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-gm5b8 started at 2019-11-11 21:10:00 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:08.484: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: true, restart count 0 Nov 11 21:13:08.484: INFO: pod-configmaps-3095fd89-26c2-4c91-bc11-f89acc95a563 started at 2019-11-11 21:12:09 +0000 UTC (0+3 container statuses recorded) Nov 11 21:13:08.484: INFO: Container createcm-volume-test ready: false, restart count 0 Nov 11 21:13:08.484: INFO: Container delcm-volume-test ready: false, restart count 0 Nov 11 21:13:08.484: INFO: Container updcm-volume-test ready: false, restart count 0 Nov 11 21:13:08.484: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-v8m7z started at 2019-11-11 21:09:50 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:08.484: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: true, restart count 0 Nov 11 21:13:08.484: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-nb9gn started at 2019-11-11 21:09:52 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:08.484: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: true, restart count 0 Nov 11 21:13:08.484: INFO: cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3-s2r9q started at 2019-11-11 21:10:02 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:08.484: INFO: Container cleanup20-97cf3380-86f6-4bb0-ae1e-5608508d78a3 ready: true, restart count 0 Nov 11 21:13:08.484: INFO: webserver-dc8bd894f-gwxx7 started at 2019-11-11 21:12:35 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:08.484: INFO: Container httpd ready: false, restart count 0 Nov 11 21:13:08.484: INFO: webserver-77c9474f75-vmx9l started at 2019-11-11 21:12:38 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:08.484: INFO: Container httpd ready: false, restart count 0 Nov 11 21:13:08.484: INFO: rs-gkz92 started at 2019-11-11 21:10:52 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:08.484: INFO: Container busybox ready: true, restart count 0 Nov 11 21:13:08.484: INFO: webserver-6f55b98-kphlw started at 2019-11-11 21:12:18 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:08.484: INFO: Container httpd ready: false, restart count 0 Nov 11 21:13:08.484: INFO: ss2-2 started at 2019-11-11 21:06:58 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:08.484: INFO: Container webserver ready: false, restart count 0 Nov 11 21:13:08.484: INFO: kube-proxy-sgvk2 started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:08.484: INFO: Container kube-proxy ready: true, restart count 0 W1111 21:13:08.542626 93370 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:13:08.846: INFO: Latency metrics for node k8s-agentpool-35271922-vmss000001 Nov 11 21:13:08.846: INFO: Logging node info for node k8s-master-35271922-vmss000000 Nov 11 21:13:08.902: INFO: Node Info: &Node{ObjectMeta:{k8s-master-35271922-vmss000000 /api/v1/nodes/k8s-master-35271922-vmss000000 f042d5b7-5ee8-4528-ae46-c997fef88c1c 41778 0 2019-11-11 20:48:07 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-35271922-vmss000000 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-35271922-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284363264 0} {<nil>} 7113636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6497931264 0} {<nil>} 6345636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:13:01 +0000 UTC,LastTransitionTime:2019-11-11 20:48:04 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:13:01 +0000 UTC,LastTransitionTime:2019-11-11 20:48:04 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:13:01 +0000 UTC,LastTransitionTime:2019-11-11 20:48:04 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:13:01 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.4,},NodeAddress{Type:Hostname,Address:k8s-master-35271922-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:8843d4f5cd0544d08ae501580db6a3a7,SystemUUID:41DCC090-746D-9E44-8888-8A77F0E0C51C,BootID:42d9eb02-ab49-4c2a-8791-7c5b2ac8f2bc,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:45e5b123e6188348715630acd2f7c1469c5b5181078ce90dc6f5399be4c17be1 k8sprow.azurecr.io/azure-cloud-controller-manager:1193984044413489154-9318bf1d],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:13:08.903: INFO: Logging kubelet events for node k8s-master-35271922-vmss000000 Nov 11 21:13:08.966: INFO: Logging pods the kubelet thinks is on node k8s-master-35271922-vmss000000 Nov 11 21:13:09.055: INFO: kube-scheduler-k8s-master-35271922-vmss000000 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:09.055: INFO: Container kube-scheduler ready: true, restart count 0 Nov 11 21:13:09.055: INFO: azure-ip-masq-agent-k7l5b started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:09.055: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:13:09.055: INFO: kube-proxy-8hzbt started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:09.055: INFO: Container kube-proxy ready: true, restart count 0 Nov 11 21:13:09.055: INFO: cloud-controller-manager-k8s-master-35271922-vmss000000 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:09.055: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 11 21:13:09.055: INFO: kube-addon-manager-k8s-master-35271922-vmss000000 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:09.055: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 11 21:13:09.055: INFO: kube-apiserver-k8s-master-35271922-vmss000000 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:09.055: INFO: Container kube-apiserver ready: true, restart count 0 Nov 11 21:13:09.056: INFO: kube-controller-manager-k8s-master-35271922-vmss000000 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:09.056: INFO: Container kube-controller-manager ready: true, restart count 0 W1111 21:13:09.114568 93370 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:13:09.265: INFO: Latency metrics for node k8s-master-35271922-vmss000000 Nov 11 21:13:09.265: INFO: Logging node info for node k8s-master-35271922-vmss000001 Nov 11 21:13:09.323: INFO: Node Info: &Node{ObjectMeta:{k8s-master-35271922-vmss000001 /api/v1/nodes/k8s-master-35271922-vmss000001 0ff06487-2e4c-4c07-a6f2-d77d1a47bbcd 41814 0 2019-11-11 20:48:39 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-35271922-vmss000001 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.6.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-35271922-vmss/virtualMachines/1,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.6.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284363264 0} {<nil>} 7113636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6497931264 0} {<nil>} 6345636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:49:01 +0000 UTC,LastTransitionTime:2019-11-11 20:49:01 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:13:04 +0000 UTC,LastTransitionTime:2019-11-11 20:48:32 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:13:04 +0000 UTC,LastTransitionTime:2019-11-11 20:48:32 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:13:04 +0000 UTC,LastTransitionTime:2019-11-11 20:48:32 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:13:04 +0000 UTC,LastTransitionTime:2019-11-11 20:48:39 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.5,},NodeAddress{Type:Hostname,Address:k8s-master-35271922-vmss000001,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:24b17940193c48e38f59d10d619fb718,SystemUUID:25FDA250-5862-9148-98C8-17FFDD874AEC,BootID:cdd7987a-6ece-464d-a177-21f53ae0b68b,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:45e5b123e6188348715630acd2f7c1469c5b5181078ce90dc6f5399be4c17be1 k8sprow.azurecr.io/azure-cloud-controller-manager:1193984044413489154-9318bf1d],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:13:09.323: INFO: Logging kubelet events for node k8s-master-35271922-vmss000001 Nov 11 21:13:09.385: INFO: Logging pods the kubelet thinks is on node k8s-master-35271922-vmss000001 Nov 11 21:13:09.482: INFO: kube-controller-manager-k8s-master-35271922-vmss000001 started at 2019-11-11 20:48:26 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:09.482: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 11 21:13:09.482: INFO: kube-scheduler-k8s-master-35271922-vmss000001 started at 2019-11-11 20:48:26 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:09.482: INFO: Container kube-scheduler ready: true, restart count 0 Nov 11 21:13:09.482: INFO: azure-ip-masq-agent-bm2t5 started at 2019-11-11 20:48:40 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:09.482: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:13:09.482: INFO: kube-proxy-5fcm6 started at 2019-11-11 20:48:40 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:09.482: INFO: Container kube-proxy ready: true, restart count 0 Nov 11 21:13:09.482: INFO: cloud-controller-manager-k8s-master-35271922-vmss000001 started at 2019-11-11 20:48:26 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:09.482: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 11 21:13:09.482: INFO: kube-addon-manager-k8s-master-35271922-vmss000001 started at 2019-11-11 20:48:26 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:09.482: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 11 21:13:09.482: INFO: kube-apiserver-k8s-master-35271922-vmss000001 started at 2019-11-11 20:48:26 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:09.482: INFO: Container kube-apiserver ready: true, restart count 0 W1111 21:13:09.539238 93370 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:13:09.667: INFO: Latency metrics for node k8s-master-35271922-vmss000001 Nov 11 21:13:09.667: INFO: Logging node info for node k8s-master-35271922-vmss000002 Nov 11 21:13:09.723: INFO: Node Info: &Node{ObjectMeta:{k8s-master-35271922-vmss000002 /api/v1/nodes/k8s-master-35271922-vmss000002 7fedbabf-dce6-48d1-994b-628369e7bcb5 41799 0 2019-11-11 20:48:08 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-35271922-vmss000002 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.4.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-35271922-vmss/virtualMachines/2,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284363264 0} {<nil>} 7113636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6497931264 0} {<nil>} 6345636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:13:03 +0000 UTC,LastTransitionTime:2019-11-11 20:48:03 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:13:03 +0000 UTC,LastTransitionTime:2019-11-11 20:48:03 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:13:03 +0000 UTC,LastTransitionTime:2019-11-11 20:48:03 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:13:03 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.6,},NodeAddress{Type:Hostname,Address:k8s-master-35271922-vmss000002,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:5ecd4c723b8845b18d85d341ba062204,SystemUUID:9E772AEA-D3D7-084D-98ED-1C1A511168B4,BootID:a73a4655-81c3-42e2-9d20-ef7e3cff3664,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:45e5b123e6188348715630acd2f7c1469c5b5181078ce90dc6f5399be4c17be1 k8sprow.azurecr.io/azure-cloud-controller-manager:1193984044413489154-9318bf1d],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:13:09.724: INFO: Logging kubelet events for node k8s-master-35271922-vmss000002 Nov 11 21:13:09.784: INFO: Logging pods the kubelet thinks is on node k8s-master-35271922-vmss000002 Nov 11 21:13:09.867: INFO: kube-proxy-t2sp7 started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:09.867: INFO: Container kube-proxy ready: true, restart count 0 Nov 11 21:13:09.867: INFO: cloud-controller-manager-k8s-master-35271922-vmss000002 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:09.867: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 11 21:13:09.867: INFO: kube-addon-manager-k8s-master-35271922-vmss000002 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:09.867: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 11 21:13:09.867: INFO: kube-apiserver-k8s-master-35271922-vmss000002 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:09.867: INFO: Container kube-apiserver ready: true, restart count 0 Nov 11 21:13:09.867: INFO: kube-controller-manager-k8s-master-35271922-vmss000002 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:09.867: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 11 21:13:09.867: INFO: kube-scheduler-k8s-master-35271922-vmss000002 started at 2019-11-11 20:47:57 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:09.867: INFO: Container kube-scheduler ready: true, restart count 0 Nov 11 21:13:09.867: INFO: azure-ip-masq-agent-cmbqm started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:09.867: INFO: Container azure-ip-masq-agent ready: true, restart count 0 W1111 21:13:09.925383 93370 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:13:10.056: INFO: Latency metrics for node k8s-master-35271922-vmss000002 Nov 11 21:13:10.056: INFO: Logging node info for node k8s-master-35271922-vmss000003 Nov 11 21:13:10.113: INFO: Node Info: &Node{ObjectMeta:{k8s-master-35271922-vmss000003 /api/v1/nodes/k8s-master-35271922-vmss000003 d4d04c51-fc7b-4cdc-a1e3-69990a198568 41819 0 2019-11-11 20:48:10 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-35271922-vmss000003 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.5.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-35271922-vmss/virtualMachines/3,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.5.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284363264 0} {<nil>} 7113636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6497931264 0} {<nil>} 6345636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:13:04 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:13:04 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:13:04 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:13:04 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.7,},NodeAddress{Type:Hostname,Address:k8s-master-35271922-vmss000003,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:53117d11ac104f35951d19e4a4448f8f,SystemUUID:048B9870-F8AE-BC47-80D8-26703FED9760,BootID:fc64edd8-d9c5-4d6f-b9e7-c902146bc724,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:45e5b123e6188348715630acd2f7c1469c5b5181078ce90dc6f5399be4c17be1 k8sprow.azurecr.io/azure-cloud-controller-manager:1193984044413489154-9318bf1d],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:13:10.113: INFO: Logging kubelet events for node k8s-master-35271922-vmss000003 Nov 11 21:13:10.172: INFO: Logging pods the kubelet thinks is on node k8s-master-35271922-vmss000003 Nov 11 21:13:10.253: INFO: cloud-controller-manager-k8s-master-35271922-vmss000003 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:10.253: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 11 21:13:10.253: INFO: kube-addon-manager-k8s-master-35271922-vmss000003 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:10.253: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 11 21:13:10.253: INFO: kube-apiserver-k8s-master-35271922-vmss000003 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:10.253: INFO: Container kube-apiserver ready: true, restart count 0 Nov 11 21:13:10.253: INFO: kube-controller-manager-k8s-master-35271922-vmss000003 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:10.253: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 11 21:13:10.253: INFO: kube-scheduler-k8s-master-35271922-vmss000003 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:10.253: INFO: Container kube-scheduler ready: true, restart count 0 Nov 11 21:13:10.253: INFO: azure-ip-masq-agent-rs668 started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:10.253: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:13:10.253: INFO: kube-proxy-tqjkf started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:10.253: INFO: Container kube-proxy ready: true, restart count 0 W1111 21:13:10.312505 93370 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:13:10.446: INFO: Latency metrics for node k8s-master-35271922-vmss000003 Nov 11 21:13:10.446: INFO: Logging node info for node k8s-master-35271922-vmss000004 Nov 11 21:13:10.503: INFO: Node Info: &Node{ObjectMeta:{k8s-master-35271922-vmss000004 /api/v1/nodes/k8s-master-35271922-vmss000004 edf29ad2-2fc8-4411-ac05-e1bf8781e340 41845 0 2019-11-11 20:48:11 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-35271922-vmss000004 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-35271922-vmss/virtualMachines/4,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284363264 0} {<nil>} 7113636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6497931264 0} {<nil>} 6345636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:13:05 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:13:05 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:13:05 +0000 UTC,LastTransitionTime:2019-11-11 20:48:05 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:13:05 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.8,},NodeAddress{Type:Hostname,Address:k8s-master-35271922-vmss000004,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:3ecc95ca67e14e2e87659efc6707e3b4,SystemUUID:0649BB2F-7C03-5743-A309-9BABFF3DD11A,BootID:1256b8d0-bf8e-4490-a45d-beaf41e45a09,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:45e5b123e6188348715630acd2f7c1469c5b5181078ce90dc6f5399be4c17be1 k8sprow.azurecr.io/azure-cloud-controller-manager:1193984044413489154-9318bf1d],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:13:10.503: INFO: Logging kubelet events for node k8s-master-35271922-vmss000004 Nov 11 21:13:10.567: INFO: Logging pods the kubelet thinks is on node k8s-master-35271922-vmss000004 Nov 11 21:13:10.649: INFO: kube-proxy-zrmqq started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:10.649: INFO: Container kube-proxy ready: true, restart count 0 Nov 11 21:13:10.649: INFO: cloud-controller-manager-k8s-master-35271922-vmss000004 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:10.649: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 11 21:13:10.649: INFO: kube-addon-manager-k8s-master-35271922-vmss000004 started at 2019-11-11 20:48:19 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:10.649: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 11 21:13:10.649: INFO: kube-apiserver-k8s-master-35271922-vmss000004 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:10.649: INFO: Container kube-apiserver ready: true, restart count 0 Nov 11 21:13:10.649: INFO: kube-controller-manager-k8s-master-35271922-vmss000004 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:10.649: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 11 21:13:10.649: INFO: kube-scheduler-k8s-master-35271922-vmss000004 started at 2019-11-11 20:47:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:10.649: INFO: Container kube-scheduler ready: true, restart count 0 Nov 11 21:13:10.649: INFO: azure-ip-masq-agent-trzsm started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:13:10.649: INFO: Container azure-ip-masq-agent ready: true, restart count 0 W1111 21:13:10.707455 93370 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:13:10.846: INFO: Latency metrics for node k8s-master-35271922-vmss000004 Nov 11 21:13:10.846: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "provisioning-4690" for this suite. Nov 11 21:15:17.082: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered Nov 11 21:15:18.943: INFO: namespace provisioning-4690 deletion completed in 2m8.039741711s
Find pod-subpath-test-csi-hostpath-dynamicpv-ns5x mentions in log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[sig\-storage\]\sCSI\sVolumes\s\[Driver\:\scsi\-hostpath\]\s\[Testpattern\:\sDynamic\sPV\s\(default\sfs\)\]\ssubPath\sshould\ssupport\sreadOnly\sdirectory\sspecified\sin\sthe\svolumeMount$'
test/e2e/storage/testsuites/subpath.go:347 Nov 11 21:24:20.314: Unexpected error: <*errors.errorString | 0xc003b5d190>: { s: "PersistentVolumeClaims [csi-hostpathp22v8] not all in phase Bound within 5m0s", } PersistentVolumeClaims [csi-hostpathp22v8] not all in phase Bound within 5m0s occurred test/e2e/storage/testsuites/base.go:366
[BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath test/e2e/storage/testsuites/base.go:93 [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath test/e2e/framework/framework.go:151 �[1mSTEP�[0m: Creating a kubernetes client Nov 11 21:19:17.186: INFO: >>> kubeConfig: /workspace/aks147280421/kubeconfig/kubeconfig.westus2.json �[1mSTEP�[0m: Building a namespace api object, basename provisioning �[1mSTEP�[0m: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in provisioning-5810 �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace [It] should support readOnly directory specified in the volumeMount test/e2e/storage/testsuites/subpath.go:347 �[1mSTEP�[0m: deploying csi-hostpath driver Nov 11 21:19:17.768: INFO: creating *v1.ServiceAccount: provisioning-5810/csi-attacher Nov 11 21:19:17.827: INFO: creating *v1.ClusterRole: external-attacher-runner-provisioning-5810 Nov 11 21:19:17.827: INFO: Define cluster role external-attacher-runner-provisioning-5810 Nov 11 21:19:17.894: INFO: creating *v1.ClusterRoleBinding: csi-attacher-role-provisioning-5810 Nov 11 21:19:17.957: INFO: creating *v1.Role: provisioning-5810/external-attacher-cfg-provisioning-5810 Nov 11 21:19:18.016: INFO: creating *v1.RoleBinding: provisioning-5810/csi-attacher-role-cfg Nov 11 21:19:18.076: INFO: creating *v1.ServiceAccount: provisioning-5810/csi-provisioner Nov 11 21:19:18.181: INFO: creating *v1.ClusterRole: external-provisioner-runner-provisioning-5810 Nov 11 21:19:18.181: INFO: Define cluster role external-provisioner-runner-provisioning-5810 Nov 11 21:19:18.249: INFO: creating *v1.ClusterRoleBinding: csi-provisioner-role-provisioning-5810 Nov 11 21:19:18.311: INFO: creating *v1.Role: provisioning-5810/external-provisioner-cfg-provisioning-5810 Nov 11 21:19:18.372: INFO: creating *v1.RoleBinding: provisioning-5810/csi-provisioner-role-cfg Nov 11 21:19:18.439: INFO: creating *v1.ServiceAccount: provisioning-5810/csi-snapshotter Nov 11 21:19:18.531: INFO: creating *v1.ClusterRole: external-snapshotter-runner-provisioning-5810 Nov 11 21:19:18.531: INFO: Define cluster role external-snapshotter-runner-provisioning-5810 Nov 11 21:19:18.590: INFO: creating *v1.ClusterRoleBinding: csi-snapshotter-role-provisioning-5810 Nov 11 21:19:18.651: INFO: creating *v1.Role: provisioning-5810/external-snapshotter-leaderelection-provisioning-5810 Nov 11 21:19:18.710: INFO: creating *v1.RoleBinding: provisioning-5810/external-snapshotter-leaderelection Nov 11 21:19:18.775: INFO: creating *v1.ServiceAccount: provisioning-5810/csi-resizer Nov 11 21:19:18.838: INFO: creating *v1.ClusterRole: external-resizer-runner-provisioning-5810 Nov 11 21:19:18.838: INFO: Define cluster role external-resizer-runner-provisioning-5810 Nov 11 21:19:18.902: INFO: creating *v1.ClusterRoleBinding: csi-resizer-role-provisioning-5810 Nov 11 21:19:18.968: INFO: creating *v1.Role: provisioning-5810/external-resizer-cfg-provisioning-5810 Nov 11 21:19:19.027: INFO: creating *v1.RoleBinding: provisioning-5810/csi-resizer-role-cfg Nov 11 21:19:19.085: INFO: creating *v1.Service: provisioning-5810/csi-hostpath-attacher Nov 11 21:19:19.169: INFO: creating *v1.StatefulSet: provisioning-5810/csi-hostpath-attacher Nov 11 21:19:19.233: INFO: creating *v1beta1.CSIDriver: csi-hostpath-provisioning-5810 Nov 11 21:19:19.293: INFO: creating *v1.Service: provisioning-5810/csi-hostpathplugin Nov 11 21:19:19.371: INFO: creating *v1.StatefulSet: provisioning-5810/csi-hostpathplugin Nov 11 21:19:19.440: INFO: creating *v1.Service: provisioning-5810/csi-hostpath-provisioner Nov 11 21:19:19.540: INFO: creating *v1.StatefulSet: provisioning-5810/csi-hostpath-provisioner Nov 11 21:19:19.607: INFO: creating *v1.Service: provisioning-5810/csi-hostpath-resizer Nov 11 21:19:19.712: INFO: creating *v1.StatefulSet: provisioning-5810/csi-hostpath-resizer Nov 11 21:19:19.779: INFO: creating *v1.Service: provisioning-5810/csi-snapshotter Nov 11 21:19:19.866: INFO: creating *v1.StatefulSet: provisioning-5810/csi-snapshotter Nov 11 21:19:19.944: INFO: creating *v1.ClusterRoleBinding: psp-csi-hostpath-role-provisioning-5810 Nov 11 21:19:20.004: INFO: Test running for native CSI Driver, not checking metrics Nov 11 21:19:20.004: INFO: Creating resource for dynamic PV �[1mSTEP�[0m: creating a StorageClass provisioning-5810-csi-hostpath-provisioning-5810-sclsk4j �[1mSTEP�[0m: creating a claim Nov 11 21:19:20.077: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil Nov 11 21:19:20.158: INFO: Waiting up to 5m0s for PersistentVolumeClaims [csi-hostpathp22v8] to have phase Bound Nov 11 21:19:20.223: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:19:22.297: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:19:24.363: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:19:26.428: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:19:28.486: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:19:30.541: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:19:32.610: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:19:34.666: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:19:36.722: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:19:38.778: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:19:40.840: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:19:42.896: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:19:44.957: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:19:47.015: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:19:49.071: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:19:51.126: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:19:53.198: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:19:55.253: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:19:57.309: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:19:59.365: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:20:01.421: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:20:03.477: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:20:05.533: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:20:07.598: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:20:09.654: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:20:11.710: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:20:13.765: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:20:15.818: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:20:17.880: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:20:19.939: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:20:21.998: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:20:24.049: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:20:26.153: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:20:28.211: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:20:30.266: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:20:32.319: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:20:34.378: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:20:36.432: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:20:38.485: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:20:40.537: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:20:42.593: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:20:44.646: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:20:46.700: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:20:48.756: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:20:50.811: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:20:52.863: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:20:54.916: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:20:56.968: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:20:59.021: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:21:01.074: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:21:03.126: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:21:05.179: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:21:07.232: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:21:09.285: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:21:11.338: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:21:13.390: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:21:15.442: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:21:17.496: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:21:19.550: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:21:21.602: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:21:23.656: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:21:25.709: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:21:27.761: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:21:29.814: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:21:31.866: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:21:33.921: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:21:35.973: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:21:38.026: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:21:40.079: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:21:42.159: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:21:44.211: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:21:46.263: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:21:48.316: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:21:50.371: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:21:52.428: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:21:54.481: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:21:56.533: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:21:58.586: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:22:00.642: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:22:02.699: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:22:04.753: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:22:06.804: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:22:08.858: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:22:10.909: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:22:12.963: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:22:15.014: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:22:17.067: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:22:19.119: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:22:21.172: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:22:23.226: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:22:25.279: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:22:27.332: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:22:29.386: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:22:31.438: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:22:33.491: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:22:35.555: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:22:37.617: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:22:39.669: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:22:41.726: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:22:43.779: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:22:45.832: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:22:47.902: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:22:49.954: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:22:52.012: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:22:54.065: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:22:56.117: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:22:58.178: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:23:00.235: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:23:02.292: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:23:04.345: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:23:06.397: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:23:08.449: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:23:10.525: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:23:12.581: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:23:14.634: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:23:16.687: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:23:18.740: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:23:20.797: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:23:22.849: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:23:24.910: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:23:26.963: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:23:29.015: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:23:31.069: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:23:33.123: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:23:35.178: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:23:37.230: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:23:39.282: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:23:41.335: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:23:43.388: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:23:45.444: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:23:47.497: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:23:49.550: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:23:51.606: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:23:53.657: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:23:55.716: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:23:57.769: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:23:59.823: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:24:01.876: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:24:03.929: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:24:05.983: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:24:08.037: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:24:10.099: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:24:12.153: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:24:14.206: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:24:16.261: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:24:18.314: INFO: PersistentVolumeClaim csi-hostpathp22v8 found but phase is Pending instead of Bound. Nov 11 21:24:20.314: FAIL: Unexpected error: <*errors.errorString | 0xc003b5d190>: { s: "PersistentVolumeClaims [csi-hostpathp22v8] not all in phase Bound within 5m0s", } PersistentVolumeClaims [csi-hostpathp22v8] not all in phase Bound within 5m0s occurred [AfterEach] [Testpattern: Dynamic PV (default fs)] subPath test/e2e/framework/framework.go:152 �[1mSTEP�[0m: Collecting events from namespace "provisioning-5810". �[1mSTEP�[0m: Found 53 events. Nov 11 21:24:20.416: INFO: At 2019-11-11 21:19:19 +0000 UTC - event for csi-hostpath-attacher: {statefulset-controller } SuccessfulCreate: create Pod csi-hostpath-attacher-0 in StatefulSet csi-hostpath-attacher successful Nov 11 21:24:20.416: INFO: At 2019-11-11 21:19:19 +0000 UTC - event for csi-hostpath-provisioner: {statefulset-controller } SuccessfulCreate: create Pod csi-hostpath-provisioner-0 in StatefulSet csi-hostpath-provisioner successful Nov 11 21:24:20.416: INFO: At 2019-11-11 21:19:19 +0000 UTC - event for csi-hostpath-resizer: {statefulset-controller } SuccessfulCreate: create Pod csi-hostpath-resizer-0 in StatefulSet csi-hostpath-resizer successful Nov 11 21:24:20.416: INFO: At 2019-11-11 21:19:19 +0000 UTC - event for csi-hostpathplugin: {statefulset-controller } SuccessfulCreate: create Pod csi-hostpathplugin-0 in StatefulSet csi-hostpathplugin successful Nov 11 21:24:20.416: INFO: At 2019-11-11 21:19:19 +0000 UTC - event for csi-snapshotter: {statefulset-controller } SuccessfulCreate: create Pod csi-snapshotter-0 in StatefulSet csi-snapshotter successful Nov 11 21:24:20.416: INFO: At 2019-11-11 21:19:20 +0000 UTC - event for csi-hostpathp22v8: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "csi-hostpath-provisioning-5810" or manually created by system administrator Nov 11 21:24:20.416: INFO: At 2019-11-11 21:19:23 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:24:20.416: INFO: At 2019-11-11 21:19:25 +0000 UTC - event for csi-hostpathplugin: {statefulset-controller } RecreatingFailedPod: StatefulSet provisioning-5810/csi-hostpathplugin is recreating failed Pod csi-hostpathplugin-0 Nov 11 21:24:20.416: INFO: At 2019-11-11 21:19:25 +0000 UTC - event for csi-hostpathplugin: {statefulset-controller } SuccessfulDelete: delete Pod csi-hostpathplugin-0 in StatefulSet csi-hostpathplugin successful Nov 11 21:24:20.416: INFO: At 2019-11-11 21:19:39 +0000 UTC - event for csi-hostpath-attacher-0: {kubelet k8s-agentpool-35271922-vmss000000} Pulling: Pulling image "quay.io/k8scsi/csi-attacher:v1.2.0" Nov 11 21:24:20.416: INFO: At 2019-11-11 21:19:39 +0000 UTC - event for csi-hostpath-provisioner-0: {kubelet k8s-agentpool-35271922-vmss000000} Pulling: Pulling image "quay.io/k8scsi/csi-provisioner:v1.4.0-rc1" Nov 11 21:24:20.416: INFO: At 2019-11-11 21:19:39 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:24:20.416: INFO: At 2019-11-11 21:19:44 +0000 UTC - event for csi-hostpath-attacher-0: {kubelet k8s-agentpool-35271922-vmss000000} Pulled: Successfully pulled image "quay.io/k8scsi/csi-attacher:v1.2.0" Nov 11 21:24:20.416: INFO: At 2019-11-11 21:19:44 +0000 UTC - event for csi-snapshotter-0: {kubelet k8s-agentpool-35271922-vmss000000} Pulling: Pulling image "quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1" Nov 11 21:24:20.416: INFO: At 2019-11-11 21:19:48 +0000 UTC - event for csi-hostpath-provisioner-0: {kubelet k8s-agentpool-35271922-vmss000000} Pulled: Successfully pulled image "quay.io/k8scsi/csi-provisioner:v1.4.0-rc1" Nov 11 21:24:20.416: INFO: At 2019-11-11 21:19:49 +0000 UTC - event for csi-hostpath-attacher-0: {kubelet k8s-agentpool-35271922-vmss000000} Created: Created container csi-attacher Nov 11 21:24:20.416: INFO: At 2019-11-11 21:19:53 +0000 UTC - event for csi-hostpath-provisioner-0: {kubelet k8s-agentpool-35271922-vmss000000} Created: Created container csi-provisioner Nov 11 21:24:20.416: INFO: At 2019-11-11 21:19:53 +0000 UTC - event for csi-snapshotter-0: {kubelet k8s-agentpool-35271922-vmss000000} Pulled: Successfully pulled image "quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1" Nov 11 21:24:20.416: INFO: At 2019-11-11 21:19:54 +0000 UTC - event for csi-hostpath-attacher-0: {kubelet k8s-agentpool-35271922-vmss000000} Started: Started container csi-attacher Nov 11 21:24:20.416: INFO: At 2019-11-11 21:19:56 +0000 UTC - event for csi-hostpath-provisioner-0: {kubelet k8s-agentpool-35271922-vmss000000} Started: Started container csi-provisioner Nov 11 21:24:20.416: INFO: At 2019-11-11 21:19:56 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:24:20.416: INFO: At 2019-11-11 21:19:56 +0000 UTC - event for csi-snapshotter-0: {kubelet k8s-agentpool-35271922-vmss000000} Created: Created container csi-snapshotter Nov 11 21:24:20.416: INFO: At 2019-11-11 21:19:57 +0000 UTC - event for csi-hostpath-resizer-0: {kubelet k8s-agentpool-35271922-vmss000000} Pulling: Pulling image "quay.io/k8scsi/csi-resizer:v0.2.0" Nov 11 21:24:20.416: INFO: At 2019-11-11 21:19:59 +0000 UTC - event for csi-hostpath-resizer-0: {kubelet k8s-agentpool-35271922-vmss000000} Pulled: Successfully pulled image "quay.io/k8scsi/csi-resizer:v0.2.0" Nov 11 21:24:20.416: INFO: At 2019-11-11 21:19:59 +0000 UTC - event for csi-snapshotter-0: {kubelet k8s-agentpool-35271922-vmss000000} Started: Started container csi-snapshotter Nov 11 21:24:20.416: INFO: At 2019-11-11 21:20:05 +0000 UTC - event for csi-hostpath-resizer-0: {kubelet k8s-agentpool-35271922-vmss000000} Created: Created container csi-resizer Nov 11 21:24:20.416: INFO: At 2019-11-11 21:20:09 +0000 UTC - event for csi-hostpath-resizer-0: {kubelet k8s-agentpool-35271922-vmss000000} Started: Started container csi-resizer Nov 11 21:24:20.416: INFO: At 2019-11-11 21:20:09 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:24:20.416: INFO: At 2019-11-11 21:20:28 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:24:20.416: INFO: At 2019-11-11 21:20:52 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:24:20.416: INFO: At 2019-11-11 21:21:02 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:24:20.416: INFO: At 2019-11-11 21:21:19 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:24:20.416: INFO: At 2019-11-11 21:21:44 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:24:20.416: INFO: At 2019-11-11 21:21:54 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:24:20.416: INFO: At 2019-11-11 21:22:02 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:24:20.416: INFO: At 2019-11-11 21:22:15 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:24:20.416: INFO: At 2019-11-11 21:22:19 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:24:20.416: INFO: At 2019-11-11 21:22:34 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:24:20.416: INFO: At 2019-11-11 21:22:41 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:24:20.416: INFO: At 2019-11-11 21:22:51 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:24:20.416: INFO: At 2019-11-11 21:23:03 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:24:20.417: INFO: At 2019-11-11 21:23:08 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:24:20.417: INFO: At 2019-11-11 21:23:22 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:24:20.417: INFO: At 2019-11-11 21:23:33 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:24:20.417: INFO: At 2019-11-11 21:23:40 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:24:20.417: INFO: At 2019-11-11 21:23:51 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:24:20.417: INFO: At 2019-11-11 21:23:56 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:24:20.417: INFO: At 2019-11-11 21:23:57 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:24:20.417: INFO: At 2019-11-11 21:24:05 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:24:20.417: INFO: At 2019-11-11 21:24:07 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:24:20.417: INFO: At 2019-11-11 21:24:09 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:24:20.417: INFO: At 2019-11-11 21:24:13 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:24:20.417: INFO: At 2019-11-11 21:24:18 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-35271922-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 11 21:24:20.479: INFO: POD NODE PHASE GRACE CONDITIONS Nov 11 21:24:20.479: INFO: csi-hostpath-attacher-0 k8s-agentpool-35271922-vmss000000 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:19:23 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:19:58 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:19:58 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:19:23 +0000 UTC }] Nov 11 21:24:20.479: INFO: csi-hostpath-provisioner-0 k8s-agentpool-35271922-vmss000000 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:19:23 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:19:59 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:19:59 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:19:23 +0000 UTC }] Nov 11 21:24:20.479: INFO: csi-hostpath-resizer-0 k8s-agentpool-35271922-vmss000000 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:19:24 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:20:13 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:20:13 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:19:24 +0000 UTC }] Nov 11 21:24:20.479: INFO: csi-hostpathplugin-0 k8s-agentpool-35271922-vmss000000 Pending [] Nov 11 21:24:20.479: INFO: csi-snapshotter-0 k8s-agentpool-35271922-vmss000000 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:19:24 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:20:20 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:20:20 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-11 21:19:24 +0000 UTC }] Nov 11 21:24:20.479: INFO: Nov 11 21:24:20.636: INFO: Logging node info for node k8s-agentpool-35271922-vmss000000 Nov 11 21:24:20.688: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool-35271922-vmss000000 /api/v1/nodes/k8s-agentpool-35271922-vmss000000 04da751e-f23a-470e-b29d-e591113d7271 54496 0 2019-11-11 20:48:07 +0000 UTC <nil> <nil> map[agentpool:agentpool beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool-35271922-vmss000000 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: storageprofile:managed storagetier:Premium_LRS] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-2030":"k8s-agentpool-35271922-vmss000000","csi-hostpath-ephemeral-7436":"k8s-agentpool-35271922-vmss000000","csi-hostpath-provisioning-239":"k8s-agentpool-35271922-vmss000000","csi-hostpath-provisioning-3597":"k8s-agentpool-35271922-vmss000000","csi-hostpath-provisioning-9334":"k8s-agentpool-35271922-vmss000000","csi-hostpath-volume-5092":"k8s-agentpool-35271922-vmss000000","csi-hostpath-volume-expand-1449":"k8s-agentpool-35271922-vmss000000","csi-hostpath-volume-expand-5724":"k8s-agentpool-35271922-vmss000000","csi-hostpath-volume-expand-5832":"k8s-agentpool-35271922-vmss000000","csi-mock-csi-mock-volumes-1804":"csi-mock-csi-mock-volumes-1804","csi-mock-csi-mock-volumes-1829":"csi-mock-csi-mock-volumes-1829","csi-mock-csi-mock-volumes-2481":"csi-mock-csi-mock-volumes-2481","csi-mock-csi-mock-volumes-3755":"csi-mock-csi-mock-volumes-3755","csi-mock-csi-mock-volumes-763":"csi-mock-csi-mock-volumes-763","csi-mock-csi-mock-volumes-9721":"csi-mock-csi-mock-volumes-9721"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-35271922-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16797044736 0} {<nil>} 16403364Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16010612736 0} {<nil>} 15635364Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:23:55 +0000 UTC,LastTransitionTime:2019-11-11 20:47:58 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:23:55 +0000 UTC,LastTransitionTime:2019-11-11 20:47:58 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:23:55 +0000 UTC,LastTransitionTime:2019-11-11 20:47:58 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:23:55 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.248.0.4,},NodeAddress{Type:Hostname,Address:k8s-agentpool-35271922-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:84c06f3ee41a47aaa558c81f34711895,SystemUUID:9BB24974-637B-EE4C-A221-0F74BB743424,BootID:59f7eb0c-c09c-4eae-a14b-0710f9e14004,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[quay.io/kubernetes_incubator/nfs-provisioner@sha256:df762117e3c891f2d2ddff46ecb0776ba1f9f3c44cfd7739b0683bcd7a7954a8 quay.io/kubernetes_incubator/nfs-provisioner:v2.2.2],SizeBytes:391772778,},ContainerImage{Names:[gluster/glusterdynamic-provisioner@sha256:90067cb05a7d217651e84576935934fd9dcff8e6bcdcbaa416bbf36fcd09dbd1 gluster/glusterdynamic-provisioner:v1.0],SizeBytes:373281573,},ContainerImage{Names:[gcr.io/google-samples/gb-frontend@sha256:35cb427341429fac3df10ff74600ea73e8ec0754d78f9ce89e0b4f3d70d53ba6 gcr.io/google-samples/gb-frontend:v6],SizeBytes:373099368,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/volume/nfs@sha256:c2ad734346f608a5f7d69cfded93c4e8094069320657bd372d12ba21dea3ea71 gcr.io/kubernetes-e2e-test-images/volume/nfs:1.0],SizeBytes:225358913,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/jessie-dnsutils@sha256:ad583e33cb284f7ef046673809b146ec4053cda19b54a85d2b180a86169715eb gcr.io/kubernetes-e2e-test-images/jessie-dnsutils:1.0],SizeBytes:195659796,},ContainerImage{Names:[httpd@sha256:addd70e4ee83f3bc9a4c1c7c41e37927ba47faf639312fc936df3afad7926f5a httpd:2.4.39-alpine],SizeBytes:126894770,},ContainerImage{Names:[httpd@sha256:eb8ccf084cf3e80eece1add239effefd171eb39adbc154d33c14260d905d4060 httpd:2.4.38-alpine],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1],SizeBytes:121711221,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727 gcr.io/kubernetes-e2e-test-images/agnhost:2.6],SizeBytes:57345321,},ContainerImage{Names:[quay.io/k8scsi/csi-provisioner@sha256:0efcb424f1dde9b9fb11a1a14f2e48ab47e1c3f08bc3a929990dcfcb1f7ab34f quay.io/k8scsi/csi-provisioner:v1.4.0-rc1],SizeBytes:54431016,},ContainerImage{Names:[quay.io/k8scsi/csi-snapshotter@sha256:e3d3e742e32d00488fdb401045b9b1d033d7ca0ab6e760f77b24750fc95e5f70 quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1],SizeBytes:51703561,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0],SizeBytes:46226754,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:0aba670b4d9d6b2e720bbf575d733156c676b693ca26501235444490300db838 quay.io/k8scsi/csi-attacher:v1.1.0],SizeBytes:42839085,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:7d46fb6eb8b890dc546029d1565d502b4a1d974d33625c6ee2bc7991b77fc1a1 quay.io/k8scsi/csi-resizer:v0.2.0],SizeBytes:42817100,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:f315c9042e56def3c05c6b04fe79ec9da6d39ddc557ca365a76cf35964ea08b6 quay.io/k8scsi/csi-resizer:v0.1.0],SizeBytes:42623056,},ContainerImage{Names:[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0],SizeBytes:42488424,},ContainerImage{Names:[redis@sha256:50899ea1ceed33fa03232f3ac57578a424faa1742c1ac9c7a7bdb95cdf19b858 redis:5.0.5-alpine],SizeBytes:29331594,},ContainerImage{Names:[quay.io/k8scsi/hostpathplugin@sha256:b4826e492fc1762fceaf9726f41575ca0a4567864d3d235da874818de18039de quay.io/k8scsi/hostpathplugin:v1.2.0-rc5],SizeBytes:28761497,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7],SizeBytes:22909487,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/echoserver@sha256:e9ba514b896cdf559eef8788b66c2c3ee55f3572df617647b4b0d8b6bf81cf19 gcr.io/kubernetes-e2e-test-images/echoserver:2.2],SizeBytes:21692741,},ContainerImage{Names:[quay.io/k8scsi/mock-driver@sha256:e0eed916b7d970bad2b7d9875f9ad16932f987f0f3d91ec5d86da68b0b5cc9d1 quay.io/k8scsi/mock-driver:v2.1.0],SizeBytes:16226335,},ContainerImage{Names:[nginx@sha256:485b610fefec7ff6c463ced9623314a04ed67e3945b9c08d7e53a47f6d108dc7 nginx:1.14-alpine],SizeBytes:16032814,},ContainerImage{Names:[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0],SizeBytes:15815995,},ContainerImage{Names:[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0],SizeBytes:14967303,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/dnsutils@sha256:2abeee84efb79c14d731966e034af33bf324d3b26ca28497555511ff094b3ddd gcr.io/kubernetes-e2e-test-images/dnsutils:1.1],SizeBytes:9349974,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nautilus@sha256:33a732d4c42a266912a5091598a0f07653c9134db4b8d571690d8afd509e0bfc gcr.io/kubernetes-e2e-test-images/nautilus:1.0],SizeBytes:4753501,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/kitten@sha256:bcbc4875c982ab39aa7c4f6acf4a287f604e996d9f34a3fbda8c3d1a7457d1f6 gcr.io/kubernetes-e2e-test-images/kitten:1.0],SizeBytes:4747037,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/test-webserver@sha256:7f93d6e32798ff28bc6289254d0c2867fe2c849c8e46edc50f8624734309812e gcr.io/kubernetes-e2e-test-images/test-webserver:1.0],SizeBytes:4732240,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 gcr.io/kubernetes-e2e-test-images/mounttest:1.0],SizeBytes:1563521,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29],SizeBytes:1154361,},ContainerImage{Names:[busybox@sha256:bbc3a03235220b170ba48a157dd097dd1379299370e1ed99ce976df0355d24f0 busybox:1.27],SizeBytes:1129289,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea k8s.gcr.io/pause-amd64:3.1 k8s.gcr.io/pause:3.1],SizeBytes:742472,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-provisioning-239^65c5c89d-04c9-11ea-a9c8-000d3af7321e],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-provisioning-239^65c5c89d-04c9-11ea-a9c8-000d3af7321e,DevicePath:,},},Config:nil,},} Nov 11 21:24:20.688: INFO: Logging kubelet events for node k8s-agentpool-35271922-vmss000000 Nov 11 21:24:20.748: INFO: Logging pods the kubelet thinks is on node k8s-agentpool-35271922-vmss000000 Nov 11 21:24:20.857: INFO: pod-subpath-test-local-preprovisionedpv-gjhc started at 2019-11-11 21:23:16 +0000 UTC (2+2 container statuses recorded) Nov 11 21:24:20.857: INFO: Init container init-volume-local-preprovisionedpv-gjhc ready: true, restart count 0 Nov 11 21:24:20.857: INFO: Init container test-init-subpath-local-preprovisionedpv-gjhc ready: true, restart count 0 Nov 11 21:24:20.857: INFO: Container test-container-subpath-local-preprovisionedpv-gjhc ready: false, restart count 0 Nov 11 21:24:20.857: INFO: Container test-container-volume-local-preprovisionedpv-gjhc ready: false, restart count 0 Nov 11 21:24:20.857: INFO: csi-hostpath-provisioner-0 started at 2019-11-11 21:19:23 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:20.857: INFO: Container csi-provisioner ready: true, restart count 0 Nov 11 21:24:20.857: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:22:41 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:20.857: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:24:20.857: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:24:09 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:20.857: INFO: Container agnhost ready: false, restart count 0 Nov 11 21:24:20.857: INFO: kube-proxy-nwdpz started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:20.857: INFO: Container kube-proxy ready: true, restart count 0 Nov 11 21:24:20.857: INFO: csi-hostpath-attacher-0 started at 2019-11-11 21:14:46 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:20.857: INFO: Container csi-attacher ready: true, restart count 0 Nov 11 21:24:20.857: INFO: csi-hostpath-resizer-0 started at 2019-11-11 21:19:24 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:20.857: INFO: Container csi-resizer ready: true, restart count 0 Nov 11 21:24:20.857: INFO: azure-ip-masq-agent-c4mb2 started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:20.857: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:24:20.857: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:22:36 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:20.857: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:24:20.857: INFO: csi-hostpath-resizer-0 started at 2019-11-11 21:14:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:20.857: INFO: Container csi-resizer ready: true, restart count 0 Nov 11 21:24:20.857: INFO: security-context-4bca757c-e658-45a9-a166-4a310da125b5 started at 2019-11-11 21:24:06 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:20.857: INFO: Container write-pod ready: false, restart count 0 Nov 11 21:24:20.857: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:20:33 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:20.857: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:24:20.857: INFO: local-client started at 2019-11-11 21:22:55 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:20.857: INFO: Container local-client ready: false, restart count 0 Nov 11 21:24:20.857: INFO: security-context-fc823105-2ca3-4045-883e-38bc1b4da1d1 started at 2019-11-11 21:23:13 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:20.857: INFO: Container write-pod ready: true, restart count 0 Nov 11 21:24:20.857: INFO: keyvault-flexvolume-zzp7p started at 2019-11-11 20:48:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:20.857: INFO: Container keyvault-flexvolume ready: true, restart count 0 Nov 11 21:24:20.857: INFO: csi-snapshotter-0 started at 2019-11-11 21:19:24 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:20.858: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 11 21:24:20.858: INFO: hostexec-k8s-agentpool-35271922-vmss000000 started at 2019-11-11 21:22:03 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:20.858: INFO: Container agnhost ready: true, restart count 0 Nov 11 21:24:20.858: INFO: csi-hostpath-provisioner-0 started at 2019-11-11 21:14:50 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:20.858: INFO: Container csi-provisioner ready: true, restart count 0 Nov 11 21:24:20.858: INFO: kubernetes-dashboard-65966766b9-lv2x8 started at 2019-11-11 20:48:59 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:20.858: INFO: Container kubernetes-dashboard ready: true, restart count 0 Nov 11 21:24:20.858: INFO: csi-hostpathplugin-0 started at 2019-11-11 21:24:18 +0000 UTC (0+0 container statuses recorded) Nov 11 21:24:20.858: INFO: csi-snapshotter-0 started at 2019-11-11 21:14:52 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:20.858: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 11 21:24:20.858: INFO: security-context-499bab7c-5a6e-4429-bb4d-c8704ec76e42 started at 2019-11-11 21:23:28 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:20.858: INFO: Container write-pod ready: true, restart count 0 Nov 11 21:24:20.858: INFO: csi-hostpath-resizer-0 started at 2019-11-11 21:19:42 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:20.858: INFO: Container csi-resizer ready: false, restart count 0 Nov 11 21:24:20.858: INFO: netserver-1 started at 2019-11-11 21:24:13 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:20.858: INFO: Container webserver ready: false, restart count 0 Nov 11 21:24:20.858: INFO: inline-volume-tester-v8qbq started at 2019-11-11 21:14:53 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:20.858: INFO: Container csi-volume-tester ready: false, restart count 0 Nov 11 21:24:20.858: INFO: csi-hostpathplugin-0 started at 2019-11-11 21:23:53 +0000 UTC (0+3 container statuses recorded) Nov 11 21:24:20.858: INFO: Container hostpath ready: false, restart count 0 Nov 11 21:24:20.858: INFO: Container liveness-probe ready: false, restart count 0 Nov 11 21:24:20.858: INFO: Container node-driver-registrar ready: false, restart count 0 Nov 11 21:24:20.858: INFO: csi-hostpath-resizer-0 started at 2019-11-11 21:18:13 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:20.858: INFO: Container csi-resizer ready: false, restart count 0 Nov 11 21:24:20.858: INFO: blobfuse-flexvol-installer-dmnhw started at 2019-11-11 20:48:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:20.858: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 Nov 11 21:24:20.858: INFO: coredns-87f5d796-6mz2l started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:20.858: INFO: Container coredns ready: true, restart count 0 Nov 11 21:24:20.858: INFO: hostpath-symlink-prep-provisioning-6202 started at 2019-11-11 21:24:02 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:20.858: INFO: Container init-volume-provisioning-6202 ready: false, restart count 0 Nov 11 21:24:20.858: INFO: csi-hostpath-attacher-0 started at 2019-11-11 21:19:23 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:20.858: INFO: Container csi-attacher ready: true, restart count 0 W1111 21:24:20.912655 93351 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:24:22.814: INFO: Latency metrics for node k8s-agentpool-35271922-vmss000000 Nov 11 21:24:22.814: INFO: Logging node info for node k8s-agentpool-35271922-vmss000001 Nov 11 21:24:22.867: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool-35271922-vmss000001 /api/v1/nodes/k8s-agentpool-35271922-vmss000001 af5ba0b6-4f7f-4468-9d3c-b3e116ca81ca 54455 0 2019-11-11 20:48:13 +0000 UTC <nil> <nil> map[agentpool:agentpool beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool-35271922-vmss000001 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: storageprofile:managed storagetier:Premium_LRS] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-8406":"k8s-agentpool-35271922-vmss000001","csi-hostpath-provisioning-2466":"k8s-agentpool-35271922-vmss000001","csi-hostpath-provisioning-265":"k8s-agentpool-35271922-vmss000001","csi-hostpath-provisioning-310":"k8s-agentpool-35271922-vmss000001","csi-hostpath-provisioning-753":"k8s-agentpool-35271922-vmss000001","csi-hostpath-volume-expand-3983":"k8s-agentpool-35271922-vmss000001","csi-mock-csi-mock-volumes-2781":"csi-mock-csi-mock-volumes-2781","csi-mock-csi-mock-volumes-2838":"csi-mock-csi-mock-volumes-2838","csi-mock-csi-mock-volumes-3306":"csi-mock-csi-mock-volumes-3306","csi-mock-csi-mock-volumes-6769":"csi-mock-csi-mock-volumes-6769","csi-mock-csi-mock-volumes-694":"csi-mock-csi-mock-volumes-694","csi-mock-csi-mock-volumes-797":"csi-mock-csi-mock-volumes-797"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-35271922-vmss/virtualMachines/1,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16797044736 0} {<nil>} 16403364Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16010612736 0} {<nil>} 15635364Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:23:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:13 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:23:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:13 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:23:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:13 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:23:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.248.0.5,},NodeAddress{Type:Hostname,Address:k8s-agentpool-35271922-vmss000001,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:1df8c22200b84e64b76150d232ecb929,SystemUUID:B03EE801-7A2D-7140-BE20-B2C844575A73,BootID:d32206f7-48b0-46c5-a542-39721566b456,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[quay.io/kubernetes_incubator/nfs-provisioner@sha256:df762117e3c891f2d2ddff46ecb0776ba1f9f3c44cfd7739b0683bcd7a7954a8 quay.io/kubernetes_incubator/nfs-provisioner:v2.2.2],SizeBytes:391772778,},ContainerImage{Names:[gcr.io/google-samples/gb-frontend@sha256:35cb427341429fac3df10ff74600ea73e8ec0754d78f9ce89e0b4f3d70d53ba6 gcr.io/google-samples/gb-frontend:v6],SizeBytes:373099368,},ContainerImage{Names:[k8s.gcr.io/etcd@sha256:12c2c5e5731c3bcd56e6f1c05c0f9198b6f06793fa7fca2fb43aab9622dc4afa k8s.gcr.io/etcd:3.3.15],SizeBytes:246640776,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/volume/nfs@sha256:c2ad734346f608a5f7d69cfded93c4e8094069320657bd372d12ba21dea3ea71 gcr.io/kubernetes-e2e-test-images/volume/nfs:1.0],SizeBytes:225358913,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/jessie-dnsutils@sha256:ad583e33cb284f7ef046673809b146ec4053cda19b54a85d2b180a86169715eb gcr.io/kubernetes-e2e-test-images/jessie-dnsutils:1.0],SizeBytes:195659796,},ContainerImage{Names:[httpd@sha256:addd70e4ee83f3bc9a4c1c7c41e37927ba47faf639312fc936df3afad7926f5a httpd:2.4.39-alpine],SizeBytes:126894770,},ContainerImage{Names:[httpd@sha256:eb8ccf084cf3e80eece1add239effefd171eb39adbc154d33c14260d905d4060 httpd:2.4.38-alpine],SizeBytes:123781643,},ContainerImage{Names:[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0],SizeBytes:71821984,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/sample-apiserver@sha256:1bafcc6fb1aa990b487850adba9cadc020e42d7905aa8a30481182a477ba24b0 gcr.io/kubernetes-e2e-test-images/sample-apiserver:1.10],SizeBytes:61365829,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727 gcr.io/kubernetes-e2e-test-images/agnhost:2.6],SizeBytes:57345321,},ContainerImage{Names:[quay.io/k8scsi/csi-provisioner@sha256:0efcb424f1dde9b9fb11a1a14f2e48ab47e1c3f08bc3a929990dcfcb1f7ab34f quay.io/k8scsi/csi-provisioner:v1.4.0-rc1],SizeBytes:54431016,},ContainerImage{Names:[quay.io/k8scsi/csi-snapshotter@sha256:e3d3e742e32d00488fdb401045b9b1d033d7ca0ab6e760f77b24750fc95e5f70 quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1],SizeBytes:51703561,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0],SizeBytes:46226754,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:0aba670b4d9d6b2e720bbf575d733156c676b693ca26501235444490300db838 quay.io/k8scsi/csi-attacher:v1.1.0],SizeBytes:42839085,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:7d46fb6eb8b890dc546029d1565d502b4a1d974d33625c6ee2bc7991b77fc1a1 quay.io/k8scsi/csi-resizer:v0.2.0],SizeBytes:42817100,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:f315c9042e56def3c05c6b04fe79ec9da6d39ddc557ca365a76cf35964ea08b6 quay.io/k8scsi/csi-resizer:v0.1.0],SizeBytes:42623056,},ContainerImage{Names:[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1],SizeBytes:42541759,},ContainerImage{Names:[gcr.io/google-containers/debian-base@sha256:6966a0aedd7592c18ff2dd803c08bd85780ee19f5e3a2e7cf908a4cd837afcde gcr.io/google-containers/debian-base:0.4.1],SizeBytes:42323657,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nonroot@sha256:d4ede5c74517090b6686219059118ed178cf4620f5db8781b32f806bb1e7395b gcr.io/kubernetes-e2e-test-images/nonroot:1.0],SizeBytes:42321438,},ContainerImage{Names:[redis@sha256:50899ea1ceed33fa03232f3ac57578a424faa1742c1ac9c7a7bdb95cdf19b858 redis:5.0.5-alpine],SizeBytes:29331594,},ContainerImage{Names:[quay.io/k8scsi/hostpathplugin@sha256:b4826e492fc1762fceaf9726f41575ca0a4567864d3d235da874818de18039de quay.io/k8scsi/hostpathplugin:v1.2.0-rc5],SizeBytes:28761497,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7],SizeBytes:22909487,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/echoserver@sha256:e9ba514b896cdf559eef8788b66c2c3ee55f3572df617647b4b0d8b6bf81cf19 gcr.io/kubernetes-e2e-test-images/echoserver:2.2],SizeBytes:21692741,},ContainerImage{Names:[quay.io/k8scsi/mock-driver@sha256:e0eed916b7d970bad2b7d9875f9ad16932f987f0f3d91ec5d86da68b0b5cc9d1 quay.io/k8scsi/mock-driver:v2.1.0],SizeBytes:16226335,},ContainerImage{Names:[nginx@sha256:485b610fefec7ff6c463ced9623314a04ed67e3945b9c08d7e53a47f6d108dc7 nginx:1.14-alpine],SizeBytes:16032814,},ContainerImage{Names:[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0],SizeBytes:15815995,},ContainerImage{Names:[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0],SizeBytes:14967303,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/dnsutils@sha256:2abeee84efb79c14d731966e034af33bf324d3b26ca28497555511ff094b3ddd gcr.io/kubernetes-e2e-test-images/dnsutils:1.1],SizeBytes:9349974,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nonewprivs@sha256:10066e9039219449fe3c81f38fe01928f87914150768ab81b62a468e51fa7411 gcr.io/kubernetes-e2e-test-images/nonewprivs:1.0],SizeBytes:6757579,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nautilus@sha256:33a732d4c42a266912a5091598a0f07653c9134db4b8d571690d8afd509e0bfc gcr.io/kubernetes-e2e-test-images/nautilus:1.0],SizeBytes:4753501,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/kitten@sha256:bcbc4875c982ab39aa7c4f6acf4a287f604e996d9f34a3fbda8c3d1a7457d1f6 gcr.io/kubernetes-e2e-test-images/kitten:1.0],SizeBytes:4747037,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/test-webserver@sha256:7f93d6e32798ff28bc6289254d0c2867fe2c849c8e46edc50f8624734309812e gcr.io/kubernetes-e2e-test-images/test-webserver:1.0],SizeBytes:4732240,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:4206620,},ContainerImage{Names:[alpine@sha256:8421d9a84432575381bfabd248f1eb56f3aa21d9d7cd2511583c68c9b7511d10 alpine:3.7],SizeBytes:4206494,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 gcr.io/kubernetes-e2e-test-images/mounttest:1.0],SizeBytes:1563521,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest-user@sha256:17319ca525ee003681fccf7e8c6b1b910ff4f49b653d939ac7f9b6e7c463933d gcr.io/kubernetes-e2e-test-images/mounttest-user:1.0],SizeBytes:1450451,},ContainerImage{Names:[busybox@sha256:1303dbf110c57f3edf68d9f5a16c082ec06c4cf7604831669faf2c712260b5a0 busybox:latest],SizeBytes:1219790,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29],SizeBytes:1154361,},ContainerImage{Names:[busybox@sha256:bbc3a03235220b170ba48a157dd097dd1379299370e1ed99ce976df0355d24f0 busybox:1.27],SizeBytes:1129289,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea k8s.gcr.io/pause-amd64:3.1 k8s.gcr.io/pause:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 11 21:24:22.868: INFO: Logging kubelet events for node k8s-agentpool-35271922-vmss000001 Nov 11 21:24:22.928: INFO: Logging pods the kubelet thinks is on node k8s-agentpool-35271922-vmss000001 Nov 11 21:24:23.060: INFO: keyvault-flexvolume-2s2wq started at 2019-11-11 20:48:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:23.060: INFO: Container keyvault-flexvolume ready: true, restart count 0 Nov 11 21:24:23.060: INFO: e2e-test-httpd-rc-6hxd6 started at 2019-11-11 21:22:06 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:23.060: INFO: Container e2e-test-httpd-rc ready: false, restart count 0 Nov 11 21:24:23.060: INFO: test-rollover-controller-lqzk7 started at 2019-11-11 21:22:47 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:23.060: INFO: Container httpd ready: true, restart count 0 Nov 11 21:24:23.060: INFO: hostexec-k8s-agentpool-35271922-vmss000001 started at 2019-11-11 21:23:45 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:23.060: INFO: Container agnhost ready: false, restart count 0 Nov 11 21:24:23.060: INFO: hostexec-k8s-agentpool-35271922-vmss000001 started at 2019-11-11 21:23:46 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:23.060: INFO: Container agnhost ready: false, restart count 0 Nov 11 21:24:23.060: INFO: test-rollover-deployment-7d7dc6548c-zzkxg started at 2019-11-11 21:24:07 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:23.060: INFO: Container redis ready: false, restart count 0 Nov 11 21:24:23.060: INFO: kube-proxy-sgvk2 started at 2019-11-11 20:48:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:23.060: INFO: Container kube-proxy ready: true, restart count 0 Nov 11 21:24:23.060: INFO: externalsvc-8rj4l started at 2019-11-11 21:22:45 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:23.060: INFO: Container externalsvc ready: true, restart count 0 Nov 11 21:24:23.060: INFO: pod-subpath-test-emptydir-x8lw started at 2019-11-11 21:23:50 +0000 UTC (2+2 container statuses recorded) Nov 11 21:24:23.060: INFO: Init container init-volume-emptydir-x8lw ready: false, restart count 0 Nov 11 21:24:23.060: INFO: Init container test-init-subpath-emptydir-x8lw ready: false, restart count 0 Nov 11 21:24:23.060: INFO: Container test-container-subpath-emptydir-x8lw ready: false, restart count 0 Nov 11 21:24:23.060: INFO: Container test-container-volume-emptydir-x8lw ready: false, restart count 0 Nov 11 21:24:23.060: INFO: csi-hostpath-attacher-0 started at 2019-11-11 21:23:26 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:23.060: INFO: Container csi-attacher ready: true, restart count 0 Nov 11 21:24:23.060: INFO: e2e-test-httpd-rc-8fca2dbdbed1f8f0dbbf980d1f6929b5-l4sb6 started at 2019-11-11 21:22:08 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:23.061: INFO: Container e2e-test-httpd-rc ready: true, restart count 0 Nov 11 21:24:23.061: INFO: csi-hostpathplugin-0 started at 2019-11-11 21:23:30 +0000 UTC (0+3 container statuses recorded) Nov 11 21:24:23.061: INFO: Container hostpath ready: false, restart count 0 Nov 11 21:24:23.061: INFO: Container liveness-probe ready: false, restart count 0 Nov 11 21:24:23.061: INFO: Container node-driver-registrar ready: false, restart count 0 Nov 11 21:24:23.061: INFO: externalsvc-vfqmb started at 2019-11-11 21:22:46 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:23.061: INFO: Container externalsvc ready: true, restart count 0 Nov 11 21:24:23.061: INFO: netserver-0 started at 2019-11-11 21:24:13 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:23.061: INFO: Container webserver ready: false, restart count 0 Nov 11 21:24:23.061: INFO: azure-ip-masq-agent-9427k started at 2019-11-11 20:48:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:23.061: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 11 21:24:23.061: INFO: sample-apiserver-deployment-8447597c78-w4btv started at 2019-11-11 21:22:17 +0000 UTC (0+2 container statuses recorded) Nov 11 21:24:23.061: INFO: Container etcd ready: true, restart count 0 Nov 11 21:24:23.061: INFO: Container sample-apiserver ready: false, restart count 0 Nov 11 21:24:23.061: INFO: csi-hostpath-provisioner-0 started at 2019-11-11 21:23:31 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:23.061: INFO: Container csi-provisioner ready: false, restart count 0 Nov 11 21:24:23.061: INFO: backofflimit-sb98q started at 2019-11-11 21:23:34 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:23.061: INFO: Container c ready: false, restart count 0 Nov 11 21:24:23.061: INFO: pod-handle-http-request started at 2019-11-11 21:24:10 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:23.061: INFO: Container pod-handle-http-request ready: false, restart count 0 Nov 11 21:24:23.061: INFO: blobfuse-flexvol-installer-tvkwl started at 2019-11-11 20:48:51 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:23.061: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 Nov 11 21:24:23.061: INFO: tiller-deploy-7559b6b885-jsnn2 started at 2019-11-11 20:48:53 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:23.061: INFO: Container tiller ready: true, restart count 0 Nov 11 21:24:23.061: INFO: busybox-3cc5f96b-0409-4e72-b3ab-eae766eb08d2 started at 2019-11-11 21:22:14 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:23.061: INFO: Container busybox ready: true, restart count 0 Nov 11 21:24:23.061: INFO: execpod96l59 started at 2019-11-11 21:24:04 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:23.061: INFO: Container agnhost-pause ready: false, restart count 0 Nov 11 21:24:23.061: INFO: csi-snapshotter-0 started at 2019-11-11 21:23:33 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:23.061: INFO: Container csi-snapshotter ready: false, restart count 0 Nov 11 21:24:23.061: INFO: metrics-server-58ff8c5ddf-thxvp started at 2019-11-11 20:48:53 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:23.061: INFO: Container metrics-server ready: true, restart count 0 Nov 11 21:24:23.061: INFO: sample-webhook-deployment-86d95b659d-5fvsq started at 2019-11-11 21:23:23 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:23.061: INFO: Container sample-webhook ready: true, restart count 0 Nov 11 21:24:23.061: INFO: csi-hostpath-resizer-0 started at 2019-11-11 21:23:32 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:23.061: INFO: Container csi-resizer ready: true, restart count 0 Nov 11 21:24:23.061: INFO: pod-submit-remove-4e755e27-38f7-4e59-a0f8-31aa18ae4fc4 started at 2019-11-11 21:22:09 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:23.061: INFO: Container nginx ready: true, restart count 0 Nov 11 21:24:23.061: INFO: backofflimit-jxgdp started at 2019-11-11 21:22:11 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:23.061: INFO: Container c ready: false, restart count 0 Nov 11 21:24:23.061: INFO: test-new-deployment-595b5b9587-w2h9h started at 2019-11-11 21:24:22 +0000 UTC (0+1 container statuses recorded) Nov 11 21:24:23.061: INFO: Container httpd ready: false, restart count 0 Nov 11 21:24:23.061: INFO: pfpod started at 2019-11-11 21:22:22 +0000 UTC (0+2 container statuses recorded) Nov 11 21:24:23.061: INFO: Container portforwardtester ready: false, restart count 0 Nov 11 21:24:23.061: INFO: Container readiness ready: false, restart count 0 W1111 21:24:23.115377 93351 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 11 21:24:33.327: INFO: Latency metrics for node k8s-agentpool-35271922-vmss000001 Nov 11 21:24:33.327: INFO: Logging node info for node k8s-master-35271922-vmss000000 Nov 11 21:24:33.379: INFO: Node Info: &Node{ObjectMeta:{k8s-master-35271922-vmss000000 /api/v1/nodes/k8s-master-35271922-vmss000000 f042d5b7-5ee8-4528-ae46-c997fef88c1c 54598 0 2019-11-11 20:48:07 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-35271922-vmss000000 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-596ce083-04bf-11ea-b31d-cec4acc7742f/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-35271922-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284363264 0} {<nil>} 7113636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6497931264 0} {<nil>} 6345636Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-11 20:48:50 +0000 UTC,LastTransitionTime:2019-11-11 20:48:50 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-11 21:24:03 +0000 UTC,LastTransitionTime:2019-11-11 20:48:04 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-11 21:24:03 +0000 UTC,LastTransitionTime:2019-11-11 20:48:04 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-11 21:24:03 +0000 UTC,LastTransitionTime:2019-11-11 20:48:04 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-11 21:24:03 +0000 UTC,LastTransitionTime:2019-11-11 20:48:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.4,},NodeAddress{Type:Hostname,Address:k8s-master-35271922-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:8843d4f5cd0544d08ae501580db6a3a7,SystemUUID:41DCC090-746D-9E44-8888-8A77F0E0C51C,BootID:42d9eb02-ab49-4c2a-8791-7c5b2ac8f2bc,KernelVersion:4.15.0-1061-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,KubeProxyVersion:v1.16.3-beta.0.56+b3cbbae08ec52a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5410d7daac5755bc6e905c6efc222ab3ffa799d32ad1a81e34c7a7d5db0f3748 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1193984044413489154-23e37f57],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:45e5b123e6188348715630acd2f7c1469c5b5181078ce90dc6f5399be4c17be1 k8sprow.azurecr.io/azure-cloud-controller-manager:1193984044413489154-9318bf1d],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa14832