Result | FAILURE |
Tests | 22 failed / 633 succeeded |
Started | |
Elapsed | 1h46m |
Revision | |
Builder | gke-prow-ssd-pool-1a225945-8wl3 |
links | {u'resultstore': {u'url': u'https://source.cloud.google.com/results/invocations/7439b986-2a46-4b93-aa44-41597612123c/targets/test'}} |
pod | 2a3ad8f8-0652-11ea-b5c7-e216f5392f03 |
resultstore | https://source.cloud.google.com/results/invocations/7439b986-2a46-4b93-aa44-41597612123c/targets/test |
infra-commit | 060550b3d |
job-version | v1.16.4-beta.0.1+d70a3ca08fe72a-dirty |
pod | 2a3ad8f8-0652-11ea-b5c7-e216f5392f03 |
repo | k8s.io/kubernetes |
repo-commit | d70a3ca08fe72ad8dd0b2d72cf032474ab2ce2a9 |
repos | {u'k8s.io/kubernetes': u'release-1.16', u'sigs.k8s.io/cloud-provider-azure': u'master'} |
revision | v1.16.4-beta.0.1+d70a3ca08fe72a-dirty |
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[k8s\.io\]\sKubelet\swhen\sscheduling\sa\sbusybox\sPod\swith\shostAliases\sshould\swrite\sentries\sto\s\/etc\/hosts\s\[LinuxOnly\]\s\[NodeConformance\]\s\[Conformance\]$'
test/e2e/framework/framework.go:152 Nov 13 21:32:56.014: Couldn't delete ns: "kubelet-test-2102": namespace kubelet-test-2102 was not deleted with limit: timed out waiting for the condition, pods remaining: 1 (&errors.errorString{s:"namespace kubelet-test-2102 was not deleted with limit: timed out waiting for the condition, pods remaining: 1"}) test/e2e/framework/framework.go:336from junit_05.xml
[BeforeEach] [k8s.io] Kubelet test/e2e/framework/framework.go:151 �[1mSTEP�[0m: Creating a kubernetes client Nov 13 21:19:25.119: INFO: >>> kubeConfig: /workspace/aks880331199/kubeconfig/kubeconfig.westus2.json �[1mSTEP�[0m: Building a namespace api object, basename kubelet-test �[1mSTEP�[0m: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in kubelet-test-2102 �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace [BeforeEach] [k8s.io] Kubelet test/e2e/common/kubelet.go:37 [It] should write entries to /etc/hosts [LinuxOnly] [NodeConformance] [Conformance] test/e2e/framework/framework.go:698 [AfterEach] [k8s.io] Kubelet test/e2e/framework/framework.go:152 Nov 13 21:22:53.853: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "kubelet-test-2102" for this suite. Nov 13 21:32:54.166: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered Nov 13 21:32:55.503: INFO: namespace: kubelet-test-2102, resource: pods, items remaining: 1 Nov 13 21:32:55.911: INFO: namespace: kubelet-test-2102, DeletionTimetamp: 2019-11-13 21:22:53 +0000 UTC, Finalizers: [kubernetes], Phase: Terminating Nov 13 21:32:55.962: INFO: namespace: kubelet-test-2102, total namespaces: 13, active: 6, terminating: 7 Nov 13 21:32:56.013: INFO: POD NODE PHASE GRACE CONDITIONS Nov 13 21:32:56.014: INFO: busybox-host-aliases7e358483-64e0-4e66-8435-049f1149ff6a k8s-agentpool-24944505-vmss000001 Running 30s [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:19:26 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:22:51 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:22:51 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:19:25 +0000 UTC }] Nov 13 21:32:56.014: INFO: Nov 13 21:32:56.014: INFO: Couldn't delete ns: "kubelet-test-2102": namespace kubelet-test-2102 was not deleted with limit: timed out waiting for the condition, pods remaining: 1 (&errors.errorString{s:"namespace kubelet-test-2102 was not deleted with limit: timed out waiting for the condition, pods remaining: 1"})
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[k8s\.io\]\sPods\sshould\ssupport\spod\sreadiness\sgates\s\[NodeFeature\:PodReadinessGate\]$'
test/e2e/common/pods.go:777 Nov 13 21:10:01.871: Unexpected error: <*errors.errorString | 0xc0000d5080>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred test/e2e/common/pods.go:811from junit_13.xml
[BeforeEach] [k8s.io] Pods test/e2e/framework/framework.go:151 �[1mSTEP�[0m: Creating a kubernetes client Nov 13 21:09:00.886: INFO: >>> kubeConfig: /workspace/aks880331199/kubeconfig/kubeconfig.westus2.json �[1mSTEP�[0m: Building a namespace api object, basename pods �[1mSTEP�[0m: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in pods-2396 �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace [BeforeEach] [k8s.io] Pods test/e2e/common/pods.go:165 [It] should support pod readiness gates [NodeFeature:PodReadinessGate] test/e2e/common/pods.go:777 �[1mSTEP�[0m: submitting the pod to kubernetes �[1mSTEP�[0m: patching pod status with condition "k8s.io/test-condition1" to true �[1mSTEP�[0m: patching pod status with condition "k8s.io/test-condition2" to true Nov 13 21:09:52.821: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Nov 13 21:09:53.818: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Nov 13 21:09:54.819: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Nov 13 21:09:55.820: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Nov 13 21:09:56.826: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Nov 13 21:09:57.819: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Nov 13 21:09:58.819: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Nov 13 21:09:59.822: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Nov 13 21:10:00.820: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Nov 13 21:10:01.819: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Nov 13 21:10:01.871: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Nov 13 21:10:01.871: FAIL: Unexpected error: <*errors.errorString | 0xc0000d5080>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred [AfterEach] [k8s.io] Pods test/e2e/framework/framework.go:152 �[1mSTEP�[0m: Collecting events from namespace "pods-2396". �[1mSTEP�[0m: Found 5 events. Nov 13 21:10:01.931: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for pod-ready: {default-scheduler } Scheduled: Successfully assigned pods-2396/pod-ready to k8s-agentpool-24944505-vmss000001 Nov 13 21:10:01.931: INFO: At 2019-11-13 21:09:21 +0000 UTC - event for pod-ready: {kubelet k8s-agentpool-24944505-vmss000001} Pulling: Pulling image "docker.io/library/busybox:1.29" Nov 13 21:10:01.931: INFO: At 2019-11-13 21:09:22 +0000 UTC - event for pod-ready: {kubelet k8s-agentpool-24944505-vmss000001} Pulled: Successfully pulled image "docker.io/library/busybox:1.29" Nov 13 21:10:01.931: INFO: At 2019-11-13 21:09:29 +0000 UTC - event for pod-ready: {kubelet k8s-agentpool-24944505-vmss000001} Created: Created container pod-readiness-gate Nov 13 21:10:01.931: INFO: At 2019-11-13 21:09:32 +0000 UTC - event for pod-ready: {kubelet k8s-agentpool-24944505-vmss000001} Started: Started container pod-readiness-gate Nov 13 21:10:02.000: INFO: POD NODE PHASE GRACE CONDITIONS Nov 13 21:10:02.000: INFO: pod-ready k8s-agentpool-24944505-vmss000001 Running [{k8s.io/test-condition2 True 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC } {k8s.io/test-condition1 True 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC } {Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:09:01 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:09:01 +0000 UTC ReadinessGatesNotReady corresponding condition of pod readiness gate "k8s.io/test-condition1" does not exist., corresponding condition of pod readiness gate "k8s.io/test-condition2" does not exist.} {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:09:40 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:09:01 +0000 UTC }] Nov 13 21:10:02.000: INFO: Nov 13 21:10:02.156: INFO: Logging node info for node k8s-agentpool-24944505-vmss000000 Nov 13 21:10:02.208: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool-24944505-vmss000000 /api/v1/nodes/k8s-agentpool-24944505-vmss000000 5bc7cf9e-80da-4e26-a2c2-20757e870d68 42147 0 2019-11-13 20:42:13 +0000 UTC <nil> <nil> map[agentpool:agentpool beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool-24944505-vmss000000 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: storageprofile:managed storagetier:Premium_LRS] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-1137":"k8s-agentpool-24944505-vmss000000","csi-hostpath-ephemeral-9135":"k8s-agentpool-24944505-vmss000000","csi-hostpath-provisioning-2585":"k8s-agentpool-24944505-vmss000000","csi-hostpath-provisioning-3303":"k8s-agentpool-24944505-vmss000000","csi-hostpath-provisioning-7066":"k8s-agentpool-24944505-vmss000000","csi-hostpath-provisioning-716":"k8s-agentpool-24944505-vmss000000","csi-hostpath-volume-9286":"k8s-agentpool-24944505-vmss000000","csi-hostpath-volume-expand-4561":"k8s-agentpool-24944505-vmss000000","csi-hostpath-volume-expand-9975":"k8s-agentpool-24944505-vmss000000","csi-mock-csi-mock-volumes-2863":"csi-mock-csi-mock-volumes-2863","csi-mock-csi-mock-volumes-5360":"csi-mock-csi-mock-volumes-5360","csi-mock-csi-mock-volumes-6481":"csi-mock-csi-mock-volumes-6481","csi-mock-csi-mock-volumes-6951":"csi-mock-csi-mock-volumes-6951","csi-mock-csi-mock-volumes-8439":"csi-mock-csi-mock-volumes-8439","csi-mock-csi-mock-volumes-8606":"csi-mock-csi-mock-volumes-8606"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-24944505-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16797569024 0} {<nil>} 16403876Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16011137024 0} {<nil>} 15635876Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 21:09:59 +0000 UTC,LastTransitionTime:2019-11-13 20:42:03 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 21:09:59 +0000 UTC,LastTransitionTime:2019-11-13 20:42:03 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 21:09:59 +0000 UTC,LastTransitionTime:2019-11-13 20:42:03 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 21:09:59 +0000 UTC,LastTransitionTime:2019-11-13 20:42:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.248.0.4,},NodeAddress{Type:Hostname,Address:k8s-agentpool-24944505-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:0a2e8938ebfc440180ed5c0b5a9edbe7,SystemUUID:8075A1F3-D764-7248-9570-A764E51D7FF2,BootID:4f2b3cbe-c61c-4c6c-93ab-18fca0df5bc5,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[quay.io/kubernetes_incubator/nfs-provisioner@sha256:df762117e3c891f2d2ddff46ecb0776ba1f9f3c44cfd7739b0683bcd7a7954a8 quay.io/kubernetes_incubator/nfs-provisioner:v2.2.2],SizeBytes:391772778,},ContainerImage{Names:[gcr.io/google-samples/gb-frontend@sha256:35cb427341429fac3df10ff74600ea73e8ec0754d78f9ce89e0b4f3d70d53ba6 gcr.io/google-samples/gb-frontend:v6],SizeBytes:373099368,},ContainerImage{Names:[k8s.gcr.io/etcd@sha256:12c2c5e5731c3bcd56e6f1c05c0f9198b6f06793fa7fca2fb43aab9622dc4afa k8s.gcr.io/etcd:3.3.15],SizeBytes:246640776,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/volume/nfs@sha256:c2ad734346f608a5f7d69cfded93c4e8094069320657bd372d12ba21dea3ea71 gcr.io/kubernetes-e2e-test-images/volume/nfs:1.0],SizeBytes:225358913,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/jessie-dnsutils@sha256:ad583e33cb284f7ef046673809b146ec4053cda19b54a85d2b180a86169715eb gcr.io/kubernetes-e2e-test-images/jessie-dnsutils:1.0],SizeBytes:195659796,},ContainerImage{Names:[httpd@sha256:addd70e4ee83f3bc9a4c1c7c41e37927ba47faf639312fc936df3afad7926f5a httpd:2.4.39-alpine],SizeBytes:126894770,},ContainerImage{Names:[httpd@sha256:eb8ccf084cf3e80eece1add239effefd171eb39adbc154d33c14260d905d4060 httpd:2.4.38-alpine],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1],SizeBytes:121711221,},ContainerImage{Names:[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0],SizeBytes:71821984,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/sample-apiserver@sha256:1bafcc6fb1aa990b487850adba9cadc020e42d7905aa8a30481182a477ba24b0 gcr.io/kubernetes-e2e-test-images/sample-apiserver:1.10],SizeBytes:61365829,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727 gcr.io/kubernetes-e2e-test-images/agnhost:2.6],SizeBytes:57345321,},ContainerImage{Names:[quay.io/k8scsi/csi-provisioner@sha256:0efcb424f1dde9b9fb11a1a14f2e48ab47e1c3f08bc3a929990dcfcb1f7ab34f quay.io/k8scsi/csi-provisioner:v1.4.0-rc1],SizeBytes:54431016,},ContainerImage{Names:[quay.io/k8scsi/csi-snapshotter@sha256:e3d3e742e32d00488fdb401045b9b1d033d7ca0ab6e760f77b24750fc95e5f70 quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1],SizeBytes:51703561,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0],SizeBytes:46226754,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:0aba670b4d9d6b2e720bbf575d733156c676b693ca26501235444490300db838 quay.io/k8scsi/csi-attacher:v1.1.0],SizeBytes:42839085,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:7d46fb6eb8b890dc546029d1565d502b4a1d974d33625c6ee2bc7991b77fc1a1 quay.io/k8scsi/csi-resizer:v0.2.0],SizeBytes:42817100,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:f315c9042e56def3c05c6b04fe79ec9da6d39ddc557ca365a76cf35964ea08b6 quay.io/k8scsi/csi-resizer:v0.1.0],SizeBytes:42623056,},ContainerImage{Names:[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1],SizeBytes:42541759,},ContainerImage{Names:[gcr.io/google-containers/debian-base@sha256:6966a0aedd7592c18ff2dd803c08bd85780ee19f5e3a2e7cf908a4cd837afcde gcr.io/google-containers/debian-base:0.4.1],SizeBytes:42323657,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nonroot@sha256:d4ede5c74517090b6686219059118ed178cf4620f5db8781b32f806bb1e7395b gcr.io/kubernetes-e2e-test-images/nonroot:1.0],SizeBytes:42321438,},ContainerImage{Names:[redis@sha256:50899ea1ceed33fa03232f3ac57578a424faa1742c1ac9c7a7bdb95cdf19b858 redis:5.0.5-alpine],SizeBytes:29331594,},ContainerImage{Names:[quay.io/k8scsi/hostpathplugin@sha256:b4826e492fc1762fceaf9726f41575ca0a4567864d3d235da874818de18039de quay.io/k8scsi/hostpathplugin:v1.2.0-rc5],SizeBytes:28761497,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7],SizeBytes:22909487,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/echoserver@sha256:e9ba514b896cdf559eef8788b66c2c3ee55f3572df617647b4b0d8b6bf81cf19 gcr.io/kubernetes-e2e-test-images/echoserver:2.2],SizeBytes:21692741,},ContainerImage{Names:[quay.io/k8scsi/mock-driver@sha256:e0eed916b7d970bad2b7d9875f9ad16932f987f0f3d91ec5d86da68b0b5cc9d1 quay.io/k8scsi/mock-driver:v2.1.0],SizeBytes:16226335,},ContainerImage{Names:[nginx@sha256:485b610fefec7ff6c463ced9623314a04ed67e3945b9c08d7e53a47f6d108dc7 nginx:1.14-alpine],SizeBytes:16032814,},ContainerImage{Names:[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0],SizeBytes:15815995,},ContainerImage{Names:[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0],SizeBytes:14967303,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/dnsutils@sha256:2abeee84efb79c14d731966e034af33bf324d3b26ca28497555511ff094b3ddd gcr.io/kubernetes-e2e-test-images/dnsutils:1.1],SizeBytes:9349974,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nonewprivs@sha256:10066e9039219449fe3c81f38fe01928f87914150768ab81b62a468e51fa7411 gcr.io/kubernetes-e2e-test-images/nonewprivs:1.0],SizeBytes:6757579,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nautilus@sha256:33a732d4c42a266912a5091598a0f07653c9134db4b8d571690d8afd509e0bfc gcr.io/kubernetes-e2e-test-images/nautilus:1.0],SizeBytes:4753501,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/kitten@sha256:bcbc4875c982ab39aa7c4f6acf4a287f604e996d9f34a3fbda8c3d1a7457d1f6 gcr.io/kubernetes-e2e-test-images/kitten:1.0],SizeBytes:4747037,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/test-webserver@sha256:7f93d6e32798ff28bc6289254d0c2867fe2c849c8e46edc50f8624734309812e gcr.io/kubernetes-e2e-test-images/test-webserver:1.0],SizeBytes:4732240,},ContainerImage{Names:[alpine@sha256:8421d9a84432575381bfabd248f1eb56f3aa21d9d7cd2511583c68c9b7511d10 alpine:3.7],SizeBytes:4206494,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 gcr.io/kubernetes-e2e-test-images/mounttest:1.0],SizeBytes:1563521,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest-user@sha256:17319ca525ee003681fccf7e8c6b1b910ff4f49b653d939ac7f9b6e7c463933d gcr.io/kubernetes-e2e-test-images/mounttest-user:1.0],SizeBytes:1450451,},ContainerImage{Names:[busybox@sha256:1303dbf110c57f3edf68d9f5a16c082ec06c4cf7604831669faf2c712260b5a0 busybox:latest],SizeBytes:1219790,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea k8s.gcr.io/pause-amd64:3.1 k8s.gcr.io/pause:3.1],SizeBytes:742472,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-volume-9286^ee731ddf-0659-11ea-a24f-000d3ac2cd71 kubernetes.io/csi/csi-mock-csi-mock-volumes-6951^4],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-volume-9286^ee731ddf-0659-11ea-a24f-000d3ac2cd71,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-6951^4,DevicePath:,},},Config:nil,},} Nov 13 21:10:02.208: INFO: Logging kubelet events for node k8s-agentpool-24944505-vmss000000 Nov 13 21:10:02.264: INFO: Logging pods the kubelet thinks is on node k8s-agentpool-24944505-vmss000000 Nov 13 21:10:02.370: INFO: local-injector started at 2019-11-13 21:09:05 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.370: INFO: Container local-injector ready: true, restart count 0 Nov 13 21:10:02.370: INFO: hostexec-k8s-agentpool-24944505-vmss000000 started at 2019-11-13 21:08:22 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.370: INFO: Container agnhost ready: true, restart count 0 Nov 13 21:10:02.370: INFO: pvc-volume-tester-2v972 started at 2019-11-13 21:09:24 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.370: INFO: Container volume-tester ready: false, restart count 0 Nov 13 21:10:02.370: INFO: csi-mockplugin-0 started at 2019-11-13 21:06:42 +0000 UTC (0+3 container statuses recorded) Nov 13 21:10:02.370: INFO: Container csi-provisioner ready: false, restart count 0 Nov 13 21:10:02.370: INFO: Container driver-registrar ready: false, restart count 0 Nov 13 21:10:02.370: INFO: Container mock ready: false, restart count 0 Nov 13 21:10:02.370: INFO: keyvault-flexvolume-sn47m started at 2019-11-13 20:42:59 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.370: INFO: Container keyvault-flexvolume ready: true, restart count 0 Nov 13 21:10:02.370: INFO: blobfuse-flexvol-installer-drhkj started at 2019-11-13 20:42:59 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.370: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 Nov 13 21:10:02.370: INFO: security-context-08f619dc-d3c7-403c-82cf-5f472eb49f93 started at 2019-11-13 21:09:00 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.370: INFO: Container write-pod ready: true, restart count 0 Nov 13 21:10:02.370: INFO: affinity-clusterip-ppxvv started at 2019-11-13 21:07:27 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.370: INFO: Container affinity-clusterip ready: true, restart count 0 Nov 13 21:10:02.370: INFO: csi-mockplugin-attacher-0 started at 2019-11-13 21:08:32 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.370: INFO: Container csi-attacher ready: true, restart count 0 Nov 13 21:10:02.370: INFO: kube-proxy-h9n4c started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.370: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 21:10:02.370: INFO: kubernetes-dashboard-65966766b9-lq4lm started at 2019-11-13 20:42:59 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.370: INFO: Container kubernetes-dashboard ready: true, restart count 0 Nov 13 21:10:02.370: INFO: hostexec-k8s-agentpool-24944505-vmss000000 started at 2019-11-13 21:07:52 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.370: INFO: Container agnhost ready: true, restart count 0 Nov 13 21:10:02.370: INFO: csi-hostpath-resizer-0 started at 2019-11-13 21:08:57 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.370: INFO: Container csi-resizer ready: true, restart count 0 Nov 13 21:10:02.370: INFO: test-rollover-deployment-7d7dc6548c-kqbhx started at 2019-11-13 21:09:13 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.370: INFO: Container redis ready: true, restart count 0 Nov 13 21:10:02.370: INFO: hostpath-injector started at 2019-11-13 21:09:52 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.370: INFO: Container hostpath-injector ready: false, restart count 0 Nov 13 21:10:02.370: INFO: hostexec-k8s-agentpool-24944505-vmss000000 started at 2019-11-13 21:09:55 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.370: INFO: Container agnhost ready: false, restart count 0 Nov 13 21:10:02.370: INFO: csi-snapshotter-0 started at 2019-11-13 21:08:58 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.370: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 13 21:10:02.370: INFO: metrics-server-58ff8c5ddf-r2d7f started at 2019-11-13 20:42:59 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.370: INFO: Container metrics-server ready: true, restart count 0 Nov 13 21:10:02.370: INFO: affinity-clusterip-zsmgp started at 2019-11-13 21:07:28 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.370: INFO: Container affinity-clusterip ready: false, restart count 0 Nov 13 21:10:02.370: INFO: azure-ip-masq-agent-qksp6 started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.370: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 21:10:02.370: INFO: tiller-deploy-7559b6b885-qxwkm started at 2019-11-13 20:42:59 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.370: INFO: Container tiller ready: true, restart count 0 Nov 13 21:10:02.370: INFO: csi-hostpath-provisioner-0 started at 2019-11-13 21:08:55 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.370: INFO: Container csi-provisioner ready: true, restart count 0 Nov 13 21:10:02.370: INFO: update-demo-nautilus-5jq22 started at 2019-11-13 21:09:18 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.370: INFO: Container update-demo ready: true, restart count 0 Nov 13 21:10:02.370: INFO: hostexec-k8s-agentpool-24944505-vmss000000 started at 2019-11-13 21:09:57 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.370: INFO: Container agnhost ready: false, restart count 0 Nov 13 21:10:02.370: INFO: pod-0 started at 2019-11-13 21:09:02 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.370: INFO: Container busybox ready: true, restart count 0 Nov 13 21:10:02.370: INFO: csi-mockplugin-0 started at 2019-11-13 21:08:32 +0000 UTC (0+3 container statuses recorded) Nov 13 21:10:02.370: INFO: Container csi-provisioner ready: true, restart count 0 Nov 13 21:10:02.370: INFO: Container driver-registrar ready: true, restart count 0 Nov 13 21:10:02.370: INFO: Container mock ready: true, restart count 0 Nov 13 21:10:02.370: INFO: csi-hostpath-attacher-0 started at 2019-11-13 21:08:50 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.370: INFO: Container csi-attacher ready: true, restart count 0 Nov 13 21:10:02.370: INFO: external-provisioner-52mff started at 2019-11-13 21:09:01 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.370: INFO: Container nfs-provisioner ready: true, restart count 0 Nov 13 21:10:02.370: INFO: nodeport-test-hhvwl started at 2019-11-13 21:09:54 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.370: INFO: Container nodeport-test ready: false, restart count 0 Nov 13 21:10:02.370: INFO: csi-hostpathplugin-0 started at 2019-11-13 21:08:53 +0000 UTC (0+3 container statuses recorded) Nov 13 21:10:02.370: INFO: Container hostpath ready: false, restart count 0 Nov 13 21:10:02.370: INFO: Container liveness-probe ready: false, restart count 0 Nov 13 21:10:02.370: INFO: Container node-driver-registrar ready: false, restart count 0 W1113 21:10:02.428077 92656 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 21:10:02.697: INFO: Latency metrics for node k8s-agentpool-24944505-vmss000000 Nov 13 21:10:02.697: INFO: Logging node info for node k8s-agentpool-24944505-vmss000001 Nov 13 21:10:02.749: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool-24944505-vmss000001 /api/v1/nodes/k8s-agentpool-24944505-vmss000001 da675d2e-0aad-4a8f-b94d-1311fb000563 42167 0 2019-11-13 20:42:24 +0000 UTC <nil> <nil> map[agentpool:agentpool beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool-24944505-vmss000001 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: storageprofile:managed storagetier:Premium_LRS] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-351":"k8s-agentpool-24944505-vmss000001","csi-hostpath-ephemeral-5584":"k8s-agentpool-24944505-vmss000001","csi-hostpath-provisioning-1638":"k8s-agentpool-24944505-vmss000001","csi-hostpath-provisioning-225":"k8s-agentpool-24944505-vmss000001","csi-hostpath-provisioning-7099":"k8s-agentpool-24944505-vmss000001","csi-hostpath-volume-6291":"k8s-agentpool-24944505-vmss000001","csi-hostpath-volume-expand-7524":"k8s-agentpool-24944505-vmss000001","csi-hostpath-volume-expand-8458":"k8s-agentpool-24944505-vmss000001","csi-mock-csi-mock-volumes-4469":"csi-mock-csi-mock-volumes-4469","csi-mock-csi-mock-volumes-4959":"csi-mock-csi-mock-volumes-4959","csi-mock-csi-mock-volumes-597":"csi-mock-csi-mock-volumes-597","csi-mock-csi-mock-volumes-8812":"csi-mock-csi-mock-volumes-8812","csi-mock-csi-mock-volumes-9088":"csi-mock-csi-mock-volumes-9088"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-24944505-vmss/virtualMachines/1,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16797569024 0} {<nil>} 16403876Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16011137024 0} {<nil>} 15635876Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 21:10:01 +0000 UTC,LastTransitionTime:2019-11-13 20:42:24 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 21:10:01 +0000 UTC,LastTransitionTime:2019-11-13 20:42:24 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 21:10:01 +0000 UTC,LastTransitionTime:2019-11-13 20:42:24 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 21:10:01 +0000 UTC,LastTransitionTime:2019-11-13 20:42:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.248.0.5,},NodeAddress{Type:Hostname,Address:k8s-agentpool-24944505-vmss000001,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:6ed41b2c4b884fccaf94748594531e06,SystemUUID:9CE5C2E3-7C34-C64A-9734-2477E19937B7,BootID:306ac3f6-a23b-423e-83c9-e3d5f3938f6c,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[quay.io/kubernetes_incubator/nfs-provisioner@sha256:df762117e3c891f2d2ddff46ecb0776ba1f9f3c44cfd7739b0683bcd7a7954a8 quay.io/kubernetes_incubator/nfs-provisioner:v2.2.2],SizeBytes:391772778,},ContainerImage{Names:[gluster/glusterdynamic-provisioner@sha256:90067cb05a7d217651e84576935934fd9dcff8e6bcdcbaa416bbf36fcd09dbd1 gluster/glusterdynamic-provisioner:v1.0],SizeBytes:373281573,},ContainerImage{Names:[gcr.io/google-samples/gb-frontend@sha256:35cb427341429fac3df10ff74600ea73e8ec0754d78f9ce89e0b4f3d70d53ba6 gcr.io/google-samples/gb-frontend:v6],SizeBytes:373099368,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/volume/nfs@sha256:c2ad734346f608a5f7d69cfded93c4e8094069320657bd372d12ba21dea3ea71 gcr.io/kubernetes-e2e-test-images/volume/nfs:1.0],SizeBytes:225358913,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/jessie-dnsutils@sha256:ad583e33cb284f7ef046673809b146ec4053cda19b54a85d2b180a86169715eb gcr.io/kubernetes-e2e-test-images/jessie-dnsutils:1.0],SizeBytes:195659796,},ContainerImage{Names:[httpd@sha256:addd70e4ee83f3bc9a4c1c7c41e37927ba47faf639312fc936df3afad7926f5a httpd:2.4.39-alpine],SizeBytes:126894770,},ContainerImage{Names:[httpd@sha256:eb8ccf084cf3e80eece1add239effefd171eb39adbc154d33c14260d905d4060 httpd:2.4.38-alpine],SizeBytes:123781643,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727 gcr.io/kubernetes-e2e-test-images/agnhost:2.6],SizeBytes:57345321,},ContainerImage{Names:[quay.io/k8scsi/csi-provisioner@sha256:0efcb424f1dde9b9fb11a1a14f2e48ab47e1c3f08bc3a929990dcfcb1f7ab34f quay.io/k8scsi/csi-provisioner:v1.4.0-rc1],SizeBytes:54431016,},ContainerImage{Names:[quay.io/k8scsi/csi-snapshotter@sha256:e3d3e742e32d00488fdb401045b9b1d033d7ca0ab6e760f77b24750fc95e5f70 quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1],SizeBytes:51703561,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0],SizeBytes:46226754,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:0aba670b4d9d6b2e720bbf575d733156c676b693ca26501235444490300db838 quay.io/k8scsi/csi-attacher:v1.1.0],SizeBytes:42839085,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:7d46fb6eb8b890dc546029d1565d502b4a1d974d33625c6ee2bc7991b77fc1a1 quay.io/k8scsi/csi-resizer:v0.2.0],SizeBytes:42817100,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:f315c9042e56def3c05c6b04fe79ec9da6d39ddc557ca365a76cf35964ea08b6 quay.io/k8scsi/csi-resizer:v0.1.0],SizeBytes:42623056,},ContainerImage{Names:[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0],SizeBytes:42488424,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nonroot@sha256:d4ede5c74517090b6686219059118ed178cf4620f5db8781b32f806bb1e7395b gcr.io/kubernetes-e2e-test-images/nonroot:1.0],SizeBytes:42321438,},ContainerImage{Names:[redis@sha256:50899ea1ceed33fa03232f3ac57578a424faa1742c1ac9c7a7bdb95cdf19b858 redis:5.0.5-alpine],SizeBytes:29331594,},ContainerImage{Names:[quay.io/k8scsi/hostpathplugin@sha256:b4826e492fc1762fceaf9726f41575ca0a4567864d3d235da874818de18039de quay.io/k8scsi/hostpathplugin:v1.2.0-rc5],SizeBytes:28761497,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7],SizeBytes:22909487,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/echoserver@sha256:e9ba514b896cdf559eef8788b66c2c3ee55f3572df617647b4b0d8b6bf81cf19 gcr.io/kubernetes-e2e-test-images/echoserver:2.2],SizeBytes:21692741,},ContainerImage{Names:[quay.io/k8scsi/mock-driver@sha256:e0eed916b7d970bad2b7d9875f9ad16932f987f0f3d91ec5d86da68b0b5cc9d1 quay.io/k8scsi/mock-driver:v2.1.0],SizeBytes:16226335,},ContainerImage{Names:[nginx@sha256:485b610fefec7ff6c463ced9623314a04ed67e3945b9c08d7e53a47f6d108dc7 nginx:1.14-alpine],SizeBytes:16032814,},ContainerImage{Names:[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0],SizeBytes:15815995,},ContainerImage{Names:[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0],SizeBytes:14967303,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/dnsutils@sha256:2abeee84efb79c14d731966e034af33bf324d3b26ca28497555511ff094b3ddd gcr.io/kubernetes-e2e-test-images/dnsutils:1.1],SizeBytes:9349974,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nautilus@sha256:33a732d4c42a266912a5091598a0f07653c9134db4b8d571690d8afd509e0bfc gcr.io/kubernetes-e2e-test-images/nautilus:1.0],SizeBytes:4753501,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/kitten@sha256:bcbc4875c982ab39aa7c4f6acf4a287f604e996d9f34a3fbda8c3d1a7457d1f6 gcr.io/kubernetes-e2e-test-images/kitten:1.0],SizeBytes:4747037,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/test-webserver@sha256:7f93d6e32798ff28bc6289254d0c2867fe2c849c8e46edc50f8624734309812e gcr.io/kubernetes-e2e-test-images/test-webserver:1.0],SizeBytes:4732240,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:4206620,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 gcr.io/kubernetes-e2e-test-images/mounttest:1.0],SizeBytes:1563521,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest-user@sha256:17319ca525ee003681fccf7e8c6b1b910ff4f49b653d939ac7f9b6e7c463933d gcr.io/kubernetes-e2e-test-images/mounttest-user:1.0],SizeBytes:1450451,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea k8s.gcr.io/pause-amd64:3.1 k8s.gcr.io/pause:3.1],SizeBytes:742472,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-volume-expand-7524^914a8876-0659-11ea-b8ce-000d3ac2cc37],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-volume-expand-7524^914a8876-0659-11ea-b8ce-000d3ac2cc37,DevicePath:,},},Config:nil,},} Nov 13 21:10:02.750: INFO: Logging kubelet events for node k8s-agentpool-24944505-vmss000001 Nov 13 21:10:02.806: INFO: Logging pods the kubelet thinks is on node k8s-agentpool-24944505-vmss000001 Nov 13 21:10:02.870: INFO: foo-q5fwr started at 2019-11-13 21:08:19 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.870: INFO: Container c ready: true, restart count 0 Nov 13 21:10:02.870: INFO: affinity-clusterip-89xtz started at 2019-11-13 21:07:28 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.870: INFO: Container affinity-clusterip ready: true, restart count 0 Nov 13 21:10:02.870: INFO: csi-hostpath-resizer-0 started at 2019-11-13 21:06:30 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.870: INFO: Container csi-resizer ready: true, restart count 0 Nov 13 21:10:02.870: INFO: security-context-c43db869-2697-4d41-bb91-f07f5d4611ef started at 2019-11-13 21:07:57 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.870: INFO: Container write-pod ready: true, restart count 0 Nov 13 21:10:02.870: INFO: foo-4hvgm started at 2019-11-13 21:08:18 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.870: INFO: Container c ready: true, restart count 0 Nov 13 21:10:02.870: INFO: all-pods-removed-45vxs started at 2019-11-13 21:08:33 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.870: INFO: Container c ready: true, restart count 0 Nov 13 21:10:02.870: INFO: test-rollover-controller-hkg5n started at 2019-11-13 21:08:11 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.870: INFO: Container httpd ready: true, restart count 0 Nov 13 21:10:02.870: INFO: pod-ready started at 2019-11-13 21:09:01 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.870: INFO: Container pod-readiness-gate ready: true, restart count 0 Nov 13 21:10:02.870: INFO: hostexec-k8s-agentpool-24944505-vmss000001 started at 2019-11-13 21:09:24 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.870: INFO: Container agnhost ready: false, restart count 0 Nov 13 21:10:02.870: INFO: csi-hostpath-attacher-0 started at 2019-11-13 21:06:25 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.870: INFO: Container csi-attacher ready: true, restart count 0 Nov 13 21:10:02.870: INFO: csi-hostpathplugin-0 started at 2019-11-13 21:06:27 +0000 UTC (0+3 container statuses recorded) Nov 13 21:10:02.870: INFO: Container hostpath ready: true, restart count 0 Nov 13 21:10:02.870: INFO: Container liveness-probe ready: true, restart count 0 Nov 13 21:10:02.870: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 13 21:10:02.870: INFO: external-provisioner-scwrw started at 2019-11-13 21:09:09 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.870: INFO: Container nfs-provisioner ready: false, restart count 0 Nov 13 21:10:02.870: INFO: pod-exec-websocket-8cfb65aa-5b86-44e9-8046-b650af7af08b started at 2019-11-13 21:09:45 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.870: INFO: Container main ready: false, restart count 0 Nov 13 21:10:02.870: INFO: pod-subpath-test-configmap-mbjl started at 2019-11-13 21:09:40 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.870: INFO: Container test-container-subpath-configmap-mbjl ready: false, restart count 0 Nov 13 21:10:02.870: INFO: pod-subpath-test-local-preprovisionedpv-dnk4 started at 2019-11-13 21:09:21 +0000 UTC (2+1 container statuses recorded) Nov 13 21:10:02.870: INFO: Init container init-volume-local-preprovisionedpv-dnk4 ready: false, restart count 0 Nov 13 21:10:02.870: INFO: Init container test-init-subpath-local-preprovisionedpv-dnk4 ready: false, restart count 0 Nov 13 21:10:02.870: INFO: Container test-container-subpath-local-preprovisionedpv-dnk4 ready: false, restart count 0 Nov 13 21:10:02.870: INFO: nodeport-test-jq77h started at 2019-11-13 21:09:54 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.870: INFO: Container nodeport-test ready: false, restart count 0 Nov 13 21:10:02.870: INFO: csi-snapshotter-0 started at 2019-11-13 21:06:31 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.870: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 13 21:10:02.870: INFO: coredns-87f5d796-jd8bc started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.870: INFO: Container coredns ready: true, restart count 0 Nov 13 21:10:02.870: INFO: hostexec-k8s-agentpool-24944505-vmss000001 started at 2019-11-13 21:09:15 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.870: INFO: Container agnhost ready: false, restart count 0 Nov 13 21:10:02.870: INFO: keyvault-flexvolume-7phpr started at 2019-11-13 20:42:58 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.870: INFO: Container keyvault-flexvolume ready: true, restart count 0 Nov 13 21:10:02.870: INFO: all-pods-removed-5cbtz started at 2019-11-13 21:08:34 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.870: INFO: Container c ready: true, restart count 0 Nov 13 21:10:02.870: INFO: csi-hostpath-provisioner-0 started at 2019-11-13 21:06:28 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.870: INFO: Container csi-provisioner ready: true, restart count 0 Nov 13 21:10:02.870: INFO: pod-subpath-test-nfs-dynamicpv-7l2d started at 2019-11-13 21:09:52 +0000 UTC (0+2 container statuses recorded) Nov 13 21:10:02.870: INFO: Container test-container-subpath-nfs-dynamicpv-7l2d ready: false, restart count 0 Nov 13 21:10:02.870: INFO: Container test-container-volume-nfs-dynamicpv-7l2d ready: false, restart count 0 Nov 13 21:10:02.870: INFO: blobfuse-flexvol-installer-cpm6h started at 2019-11-13 20:42:58 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.870: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 Nov 13 21:10:02.870: INFO: downwardapi-volume-c2c8f7dd-e0e3-41f2-ab07-e07d65199c5b started at 2019-11-13 21:09:49 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.870: INFO: Container client-container ready: false, restart count 0 Nov 13 21:10:02.870: INFO: hostexec-k8s-agentpool-24944505-vmss000001 started at 2019-11-13 21:09:57 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.870: INFO: Container agnhost ready: false, restart count 0 Nov 13 21:10:02.870: INFO: azure-ip-masq-agent-wwpg7 started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.870: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 21:10:02.870: INFO: backofflimit-vj4k9 started at 2019-11-13 21:09:16 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.870: INFO: Container c ready: false, restart count 0 Nov 13 21:10:02.870: INFO: pod-subpath-test-hostpath-cz8v started at 2019-11-13 21:09:45 +0000 UTC (2+2 container statuses recorded) Nov 13 21:10:02.870: INFO: Init container init-volume-hostpath-cz8v ready: false, restart count 0 Nov 13 21:10:02.870: INFO: Init container test-init-subpath-hostpath-cz8v ready: false, restart count 0 Nov 13 21:10:02.870: INFO: Container test-container-subpath-hostpath-cz8v ready: false, restart count 0 Nov 13 21:10:02.870: INFO: Container test-container-volume-hostpath-cz8v ready: false, restart count 0 Nov 13 21:10:02.870: INFO: kube-proxy-7bpf7 started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.870: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 21:10:02.870: INFO: hostexec-k8s-agentpool-24944505-vmss000001 started at 2019-11-13 21:08:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.870: INFO: Container agnhost ready: true, restart count 0 Nov 13 21:10:02.870: INFO: update-demo-nautilus-nnch2 started at 2019-11-13 21:09:19 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.870: INFO: Container update-demo ready: false, restart count 0 Nov 13 21:10:02.870: INFO: httpd started at 2019-11-13 21:09:55 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.870: INFO: Container httpd ready: false, restart count 0 Nov 13 21:10:02.870: INFO: labelsupdate245feb5e-c35c-4192-9e4a-200f6fa6ca7c started at 2019-11-13 21:09:47 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:02.870: INFO: Container client-container ready: false, restart count 0 W1113 21:10:02.923584 92656 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 21:10:03.356: INFO: Latency metrics for node k8s-agentpool-24944505-vmss000001 Nov 13 21:10:03.356: INFO: Logging node info for node k8s-master-24944505-vmss000000 Nov 13 21:10:03.411: INFO: Node Info: &Node{ObjectMeta:{k8s-master-24944505-vmss000000 /api/v1/nodes/k8s-master-24944505-vmss000000 048fbae2-b97f-43a2-b471-398180136f12 41110 0 2019-11-13 20:42:13 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-24944505-vmss000000 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-24944505-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284887552 0} {<nil>} 7114148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6498455552 0} {<nil>} 6346148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 21:09:07 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 21:09:07 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 21:09:07 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 21:09:07 +0000 UTC,LastTransitionTime:2019-11-13 20:42:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.4,},NodeAddress{Type:Hostname,Address:k8s-master-24944505-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:7bd754c0dc8647518c3043bcae3a76cc,SystemUUID:FA5CFE99-8148-6C46-B4E6-5EA2A5DE9D2A,BootID:74f7b0a2-e85c-438e-a642-c5b32900bb28,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:3237a53b135d8692ade82eca845fd0259ecbfcbe05fad35400e7d5c2018c4abd k8sprow.azurecr.io/azure-cloud-controller-manager:1194710255657816064-e688ecb0],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 21:10:03.411: INFO: Logging kubelet events for node k8s-master-24944505-vmss000000 Nov 13 21:10:03.467: INFO: Logging pods the kubelet thinks is on node k8s-master-24944505-vmss000000 Nov 13 21:10:03.544: INFO: kube-proxy-d9jjh started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:03.544: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 21:10:03.544: INFO: cloud-controller-manager-k8s-master-24944505-vmss000000 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:03.544: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 13 21:10:03.544: INFO: kube-addon-manager-k8s-master-24944505-vmss000000 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:03.544: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 13 21:10:03.544: INFO: kube-apiserver-k8s-master-24944505-vmss000000 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:03.544: INFO: Container kube-apiserver ready: true, restart count 0 Nov 13 21:10:03.544: INFO: kube-controller-manager-k8s-master-24944505-vmss000000 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:03.544: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 13 21:10:03.544: INFO: kube-scheduler-k8s-master-24944505-vmss000000 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:03.544: INFO: Container kube-scheduler ready: true, restart count 0 Nov 13 21:10:03.544: INFO: azure-ip-masq-agent-6pnst started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:03.544: INFO: Container azure-ip-masq-agent ready: true, restart count 0 W1113 21:10:03.606484 92656 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 21:10:03.735: INFO: Latency metrics for node k8s-master-24944505-vmss000000 Nov 13 21:10:03.735: INFO: Logging node info for node k8s-master-24944505-vmss000001 Nov 13 21:10:03.787: INFO: Node Info: &Node{ObjectMeta:{k8s-master-24944505-vmss000001 /api/v1/nodes/k8s-master-24944505-vmss000001 a2619878-b119-44f9-b89f-4073bebbb56c 41075 0 2019-11-13 20:42:40 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-24944505-vmss000001 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.6.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-24944505-vmss/virtualMachines/1,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.6.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284883456 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6498451456 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:43:08 +0000 UTC,LastTransitionTime:2019-11-13 20:43:08 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 21:09:04 +0000 UTC,LastTransitionTime:2019-11-13 20:42:34 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 21:09:04 +0000 UTC,LastTransitionTime:2019-11-13 20:42:34 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 21:09:04 +0000 UTC,LastTransitionTime:2019-11-13 20:42:34 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 21:09:04 +0000 UTC,LastTransitionTime:2019-11-13 20:42:40 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.5,},NodeAddress{Type:Hostname,Address:k8s-master-24944505-vmss000001,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:c866a7c1ed64481990416e24387dd317,SystemUUID:EBE64557-2AC8-5740-9FFE-AB9F12412FBC,BootID:3dbb0e0e-0479-4821-bc3f-a8269d250e4c,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:3237a53b135d8692ade82eca845fd0259ecbfcbe05fad35400e7d5c2018c4abd k8sprow.azurecr.io/azure-cloud-controller-manager:1194710255657816064-e688ecb0],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 21:10:03.788: INFO: Logging kubelet events for node k8s-master-24944505-vmss000001 Nov 13 21:10:03.844: INFO: Logging pods the kubelet thinks is on node k8s-master-24944505-vmss000001 Nov 13 21:10:03.917: INFO: kube-apiserver-k8s-master-24944505-vmss000001 started at 2019-11-13 20:42:28 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:03.917: INFO: Container kube-apiserver ready: true, restart count 0 Nov 13 21:10:03.917: INFO: kube-controller-manager-k8s-master-24944505-vmss000001 started at 2019-11-13 20:42:28 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:03.917: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 13 21:10:03.917: INFO: kube-proxy-w4bz5 started at 2019-11-13 20:42:41 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:03.917: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 21:10:03.917: INFO: azure-ip-masq-agent-bs52d started at 2019-11-13 20:42:41 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:03.917: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 21:10:03.917: INFO: kube-scheduler-k8s-master-24944505-vmss000001 started at 2019-11-13 20:42:28 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:03.917: INFO: Container kube-scheduler ready: true, restart count 0 Nov 13 21:10:03.917: INFO: cloud-controller-manager-k8s-master-24944505-vmss000001 started at 2019-11-13 20:42:28 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:03.917: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 13 21:10:03.917: INFO: kube-addon-manager-k8s-master-24944505-vmss000001 started at 2019-11-13 20:42:28 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:03.917: INFO: Container kube-addon-manager ready: true, restart count 0 W1113 21:10:03.991600 92656 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 21:10:04.108: INFO: Latency metrics for node k8s-master-24944505-vmss000001 Nov 13 21:10:04.108: INFO: Logging node info for node k8s-master-24944505-vmss000002 Nov 13 21:10:04.162: INFO: Node Info: &Node{ObjectMeta:{k8s-master-24944505-vmss000002 /api/v1/nodes/k8s-master-24944505-vmss000002 f3b05267-a009-4dba-9f97-4f772e48707e 41108 0 2019-11-13 20:42:13 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-24944505-vmss000002 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.4.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-24944505-vmss/virtualMachines/2,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284887552 0} {<nil>} 7114148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6498455552 0} {<nil>} 6346148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 21:09:07 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 21:09:07 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 21:09:07 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 21:09:07 +0000 UTC,LastTransitionTime:2019-11-13 20:42:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.6,},NodeAddress{Type:Hostname,Address:k8s-master-24944505-vmss000002,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:fc1777f37ced4b9c97acb9b30e276c4f,SystemUUID:E9DF19DC-D02C-714C-A7D6-0BA6CE959C55,BootID:622958a8-d45d-4edf-9266-4dbb759e2359,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:3237a53b135d8692ade82eca845fd0259ecbfcbe05fad35400e7d5c2018c4abd k8sprow.azurecr.io/azure-cloud-controller-manager:1194710255657816064-e688ecb0],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 21:10:04.162: INFO: Logging kubelet events for node k8s-master-24944505-vmss000002 Nov 13 21:10:04.219: INFO: Logging pods the kubelet thinks is on node k8s-master-24944505-vmss000002 Nov 13 21:10:04.295: INFO: kube-addon-manager-k8s-master-24944505-vmss000002 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:04.295: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 13 21:10:04.295: INFO: kube-apiserver-k8s-master-24944505-vmss000002 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:04.295: INFO: Container kube-apiserver ready: true, restart count 0 Nov 13 21:10:04.295: INFO: kube-controller-manager-k8s-master-24944505-vmss000002 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:04.295: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 13 21:10:04.295: INFO: kube-scheduler-k8s-master-24944505-vmss000002 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:04.295: INFO: Container kube-scheduler ready: true, restart count 0 Nov 13 21:10:04.295: INFO: cloud-controller-manager-k8s-master-24944505-vmss000002 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:04.295: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 13 21:10:04.295: INFO: azure-ip-masq-agent-6skf2 started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:04.295: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 21:10:04.295: INFO: kube-proxy-xwz78 started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:04.295: INFO: Container kube-proxy ready: true, restart count 0 W1113 21:10:04.350366 92656 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 21:10:04.474: INFO: Latency metrics for node k8s-master-24944505-vmss000002 Nov 13 21:10:04.474: INFO: Logging node info for node k8s-master-24944505-vmss000003 Nov 13 21:10:04.526: INFO: Node Info: &Node{ObjectMeta:{k8s-master-24944505-vmss000003 /api/v1/nodes/k8s-master-24944505-vmss000003 888cc19e-c62d-4945-a425-97d14e38a0b8 42149 0 2019-11-13 20:42:35 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-24944505-vmss000003 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.5.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-24944505-vmss/virtualMachines/3,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.5.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284887552 0} {<nil>} 7114148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6498455552 0} {<nil>} 6346148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 21:09:59 +0000 UTC,LastTransitionTime:2019-11-13 20:42:29 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 21:09:59 +0000 UTC,LastTransitionTime:2019-11-13 20:42:29 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 21:09:59 +0000 UTC,LastTransitionTime:2019-11-13 20:42:29 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 21:09:59 +0000 UTC,LastTransitionTime:2019-11-13 20:42:35 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.7,},NodeAddress{Type:Hostname,Address:k8s-master-24944505-vmss000003,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:2da5fe5a442b4e178a2b62ba91366e4b,SystemUUID:D2D1E816-D70E-3D40-B959-B00A2B76A83A,BootID:c5929aa1-3dac-45d5-bb2c-174c73c7300e,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:3237a53b135d8692ade82eca845fd0259ecbfcbe05fad35400e7d5c2018c4abd k8sprow.azurecr.io/azure-cloud-controller-manager:1194710255657816064-e688ecb0],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 21:10:04.527: INFO: Logging kubelet events for node k8s-master-24944505-vmss000003 Nov 13 21:10:04.582: INFO: Logging pods the kubelet thinks is on node k8s-master-24944505-vmss000003 Nov 13 21:10:04.673: INFO: azure-ip-masq-agent-nlqss started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:04.673: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 21:10:04.673: INFO: kube-proxy-kgh5h started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:04.673: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 21:10:04.673: INFO: kube-addon-manager-k8s-master-24944505-vmss000003 started at 2019-11-13 20:42:22 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:04.673: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 13 21:10:04.673: INFO: kube-apiserver-k8s-master-24944505-vmss000003 started at 2019-11-13 20:42:23 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:04.673: INFO: Container kube-apiserver ready: true, restart count 0 Nov 13 21:10:04.673: INFO: kube-controller-manager-k8s-master-24944505-vmss000003 started at 2019-11-13 20:42:23 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:04.673: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 13 21:10:04.673: INFO: kube-scheduler-k8s-master-24944505-vmss000003 started at 2019-11-13 20:42:23 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:04.673: INFO: Container kube-scheduler ready: true, restart count 0 Nov 13 21:10:04.673: INFO: cloud-controller-manager-k8s-master-24944505-vmss000003 started at 2019-11-13 20:42:44 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:04.673: INFO: Container cloud-controller-manager ready: true, restart count 0 W1113 21:10:04.725671 92656 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 21:10:04.849: INFO: Latency metrics for node k8s-master-24944505-vmss000003 Nov 13 21:10:04.849: INFO: Logging node info for node k8s-master-24944505-vmss000004 Nov 13 21:10:04.902: INFO: Node Info: &Node{ObjectMeta:{k8s-master-24944505-vmss000004 /api/v1/nodes/k8s-master-24944505-vmss000004 a26d844f-0612-406b-9a65-7e2ac952a2d5 41112 0 2019-11-13 20:42:14 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-24944505-vmss000004 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-24944505-vmss/virtualMachines/4,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284887552 0} {<nil>} 7114148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6498455552 0} {<nil>} 6346148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 21:09:07 +0000 UTC,LastTransitionTime:2019-11-13 20:42:09 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 21:09:07 +0000 UTC,LastTransitionTime:2019-11-13 20:42:09 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 21:09:07 +0000 UTC,LastTransitionTime:2019-11-13 20:42:09 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 21:09:07 +0000 UTC,LastTransitionTime:2019-11-13 20:42:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.8,},NodeAddress{Type:Hostname,Address:k8s-master-24944505-vmss000004,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:2208959aca1045d9b48fce8d784c1e41,SystemUUID:E253DB3A-3A4D-5648-8169-B2E2976CFB4B,BootID:09bc6101-2da9-4c8e-8274-43db8c5fe1cf,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:3237a53b135d8692ade82eca845fd0259ecbfcbe05fad35400e7d5c2018c4abd k8sprow.azurecr.io/azure-cloud-controller-manager:1194710255657816064-e688ecb0],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 21:10:04.902: INFO: Logging kubelet events for node k8s-master-24944505-vmss000004 Nov 13 21:10:04.958: INFO: Logging pods the kubelet thinks is on node k8s-master-24944505-vmss000004 Nov 13 21:10:05.035: INFO: kube-apiserver-k8s-master-24944505-vmss000004 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:05.035: INFO: Container kube-apiserver ready: true, restart count 0 Nov 13 21:10:05.035: INFO: kube-controller-manager-k8s-master-24944505-vmss000004 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:05.035: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 13 21:10:05.035: INFO: kube-scheduler-k8s-master-24944505-vmss000004 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:05.035: INFO: Container kube-scheduler ready: true, restart count 0 Nov 13 21:10:05.035: INFO: azure-ip-masq-agent-xnw2h started at 2019-11-13 20:42:38 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:05.035: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 21:10:05.035: INFO: kube-proxy-fld6p started at 2019-11-13 20:42:38 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:05.035: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 21:10:05.035: INFO: cloud-controller-manager-k8s-master-24944505-vmss000004 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:05.035: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 13 21:10:05.035: INFO: kube-addon-manager-k8s-master-24944505-vmss000004 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:10:05.035: INFO: Container kube-addon-manager ready: true, restart count 0 W1113 21:10:05.089481 92656 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 21:10:05.214: INFO: Latency metrics for node k8s-master-24944505-vmss000004 Nov 13 21:10:05.214: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "pods-2396" for this suite. Nov 13 21:12:15.439: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered Nov 13 21:12:17.403: INFO: namespace pods-2396 deletion completed in 2m12.134008051s
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[k8s\.io\]\sPods\sshould\ssupport\spod\sreadiness\sgates\s\[NodeFeature\:PodReadinessGate\]$'
test/e2e/common/pods.go:777 Nov 13 21:06:44.364: Unexpected error: <*errors.errorString | 0xc0000d5080>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred test/e2e/common/pods.go:811from junit_13.xml
[BeforeEach] [k8s.io] Pods test/e2e/framework/framework.go:151 �[1mSTEP�[0m: Creating a kubernetes client Nov 13 21:05:39.242: INFO: >>> kubeConfig: /workspace/aks880331199/kubeconfig/kubeconfig.westus2.json �[1mSTEP�[0m: Building a namespace api object, basename pods �[1mSTEP�[0m: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in pods-7301 �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace [BeforeEach] [k8s.io] Pods test/e2e/common/pods.go:165 [It] should support pod readiness gates [NodeFeature:PodReadinessGate] test/e2e/common/pods.go:777 �[1mSTEP�[0m: submitting the pod to kubernetes �[1mSTEP�[0m: patching pod status with condition "k8s.io/test-condition1" to true �[1mSTEP�[0m: patching pod status with condition "k8s.io/test-condition2" to true Nov 13 21:06:25.183: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Nov 13 21:06:26.184: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Nov 13 21:06:27.187: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Nov 13 21:06:28.184: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Nov 13 21:06:29.184: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Nov 13 21:06:30.184: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Nov 13 21:06:31.184: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Nov 13 21:06:32.184: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Nov 13 21:06:33.184: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false �[1mSTEP�[0m: patching pod status with condition "k8s.io/test-condition1" to false Nov 13 21:06:35.298: INFO: Expect the Ready condition of pod "pod-ready" to be false, but got true Nov 13 21:06:36.299: INFO: Expect the Ready condition of pod "pod-ready" to be false, but got true Nov 13 21:06:37.299: INFO: Expect the Ready condition of pod "pod-ready" to be false, but got true Nov 13 21:06:38.301: INFO: Expect the Ready condition of pod "pod-ready" to be false, but got true Nov 13 21:06:39.300: INFO: Expect the Ready condition of pod "pod-ready" to be false, but got true Nov 13 21:06:40.308: INFO: Expect the Ready condition of pod "pod-ready" to be false, but got true Nov 13 21:06:41.299: INFO: Expect the Ready condition of pod "pod-ready" to be false, but got true Nov 13 21:06:42.301: INFO: Expect the Ready condition of pod "pod-ready" to be false, but got true Nov 13 21:06:43.298: INFO: Expect the Ready condition of pod "pod-ready" to be false, but got true Nov 13 21:06:44.302: INFO: Expect the Ready condition of pod "pod-ready" to be false, but got true Nov 13 21:06:44.364: INFO: Expect the Ready condition of pod "pod-ready" to be false, but got true Nov 13 21:06:44.364: FAIL: Unexpected error: <*errors.errorString | 0xc0000d5080>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred [AfterEach] [k8s.io] Pods test/e2e/framework/framework.go:152 �[1mSTEP�[0m: Collecting events from namespace "pods-7301". �[1mSTEP�[0m: Found 5 events. Nov 13 21:06:44.417: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for pod-ready: {default-scheduler } Scheduled: Successfully assigned pods-7301/pod-ready to k8s-agentpool-24944505-vmss000001 Nov 13 21:06:44.417: INFO: At 2019-11-13 21:05:51 +0000 UTC - event for pod-ready: {kubelet k8s-agentpool-24944505-vmss000001} Pulling: Pulling image "docker.io/library/busybox:1.29" Nov 13 21:06:44.417: INFO: At 2019-11-13 21:05:54 +0000 UTC - event for pod-ready: {kubelet k8s-agentpool-24944505-vmss000001} Pulled: Successfully pulled image "docker.io/library/busybox:1.29" Nov 13 21:06:44.417: INFO: At 2019-11-13 21:05:57 +0000 UTC - event for pod-ready: {kubelet k8s-agentpool-24944505-vmss000001} Created: Created container pod-readiness-gate Nov 13 21:06:44.417: INFO: At 2019-11-13 21:06:00 +0000 UTC - event for pod-ready: {kubelet k8s-agentpool-24944505-vmss000001} Started: Started container pod-readiness-gate Nov 13 21:06:44.471: INFO: POD NODE PHASE GRACE CONDITIONS Nov 13 21:06:44.471: INFO: pod-ready k8s-agentpool-24944505-vmss000001 Running [{k8s.io/test-condition2 True 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC } {k8s.io/test-condition1 False 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC } {Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:05:41 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:06:33 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:06:06 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:05:39 +0000 UTC }] Nov 13 21:06:44.471: INFO: Nov 13 21:06:44.626: INFO: Logging node info for node k8s-agentpool-24944505-vmss000000 Nov 13 21:06:44.682: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool-24944505-vmss000000 /api/v1/nodes/k8s-agentpool-24944505-vmss000000 5bc7cf9e-80da-4e26-a2c2-20757e870d68 38387 0 2019-11-13 20:42:13 +0000 UTC <nil> <nil> map[agentpool:agentpool beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool-24944505-vmss000000 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: storageprofile:managed storagetier:Premium_LRS] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-1137":"k8s-agentpool-24944505-vmss000000","csi-hostpath-ephemeral-9135":"k8s-agentpool-24944505-vmss000000","csi-hostpath-provisioning-2585":"k8s-agentpool-24944505-vmss000000","csi-hostpath-provisioning-3303":"k8s-agentpool-24944505-vmss000000","csi-hostpath-provisioning-7066":"k8s-agentpool-24944505-vmss000000","csi-hostpath-provisioning-716":"k8s-agentpool-24944505-vmss000000","csi-hostpath-volume-expand-4561":"k8s-agentpool-24944505-vmss000000","csi-hostpath-volume-expand-9975":"k8s-agentpool-24944505-vmss000000","csi-mock-csi-mock-volumes-2863":"csi-mock-csi-mock-volumes-2863","csi-mock-csi-mock-volumes-5360":"csi-mock-csi-mock-volumes-5360","csi-mock-csi-mock-volumes-6481":"csi-mock-csi-mock-volumes-6481","csi-mock-csi-mock-volumes-8606":"csi-mock-csi-mock-volumes-8606"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-24944505-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16797569024 0} {<nil>} 16403876Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16011137024 0} {<nil>} 15635876Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 21:06:38 +0000 UTC,LastTransitionTime:2019-11-13 20:42:03 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 21:06:38 +0000 UTC,LastTransitionTime:2019-11-13 20:42:03 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 21:06:38 +0000 UTC,LastTransitionTime:2019-11-13 20:42:03 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 21:06:38 +0000 UTC,LastTransitionTime:2019-11-13 20:42:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.248.0.4,},NodeAddress{Type:Hostname,Address:k8s-agentpool-24944505-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:0a2e8938ebfc440180ed5c0b5a9edbe7,SystemUUID:8075A1F3-D764-7248-9570-A764E51D7FF2,BootID:4f2b3cbe-c61c-4c6c-93ab-18fca0df5bc5,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[quay.io/kubernetes_incubator/nfs-provisioner@sha256:df762117e3c891f2d2ddff46ecb0776ba1f9f3c44cfd7739b0683bcd7a7954a8 quay.io/kubernetes_incubator/nfs-provisioner:v2.2.2],SizeBytes:391772778,},ContainerImage{Names:[gcr.io/google-samples/gb-frontend@sha256:35cb427341429fac3df10ff74600ea73e8ec0754d78f9ce89e0b4f3d70d53ba6 gcr.io/google-samples/gb-frontend:v6],SizeBytes:373099368,},ContainerImage{Names:[k8s.gcr.io/etcd@sha256:12c2c5e5731c3bcd56e6f1c05c0f9198b6f06793fa7fca2fb43aab9622dc4afa k8s.gcr.io/etcd:3.3.15],SizeBytes:246640776,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/volume/nfs@sha256:c2ad734346f608a5f7d69cfded93c4e8094069320657bd372d12ba21dea3ea71 gcr.io/kubernetes-e2e-test-images/volume/nfs:1.0],SizeBytes:225358913,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/jessie-dnsutils@sha256:ad583e33cb284f7ef046673809b146ec4053cda19b54a85d2b180a86169715eb gcr.io/kubernetes-e2e-test-images/jessie-dnsutils:1.0],SizeBytes:195659796,},ContainerImage{Names:[httpd@sha256:addd70e4ee83f3bc9a4c1c7c41e37927ba47faf639312fc936df3afad7926f5a httpd:2.4.39-alpine],SizeBytes:126894770,},ContainerImage{Names:[httpd@sha256:eb8ccf084cf3e80eece1add239effefd171eb39adbc154d33c14260d905d4060 httpd:2.4.38-alpine],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1],SizeBytes:121711221,},ContainerImage{Names:[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0],SizeBytes:71821984,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/sample-apiserver@sha256:1bafcc6fb1aa990b487850adba9cadc020e42d7905aa8a30481182a477ba24b0 gcr.io/kubernetes-e2e-test-images/sample-apiserver:1.10],SizeBytes:61365829,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727 gcr.io/kubernetes-e2e-test-images/agnhost:2.6],SizeBytes:57345321,},ContainerImage{Names:[quay.io/k8scsi/csi-provisioner@sha256:0efcb424f1dde9b9fb11a1a14f2e48ab47e1c3f08bc3a929990dcfcb1f7ab34f quay.io/k8scsi/csi-provisioner:v1.4.0-rc1],SizeBytes:54431016,},ContainerImage{Names:[quay.io/k8scsi/csi-snapshotter@sha256:e3d3e742e32d00488fdb401045b9b1d033d7ca0ab6e760f77b24750fc95e5f70 quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1],SizeBytes:51703561,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0],SizeBytes:46226754,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:0aba670b4d9d6b2e720bbf575d733156c676b693ca26501235444490300db838 quay.io/k8scsi/csi-attacher:v1.1.0],SizeBytes:42839085,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:7d46fb6eb8b890dc546029d1565d502b4a1d974d33625c6ee2bc7991b77fc1a1 quay.io/k8scsi/csi-resizer:v0.2.0],SizeBytes:42817100,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:f315c9042e56def3c05c6b04fe79ec9da6d39ddc557ca365a76cf35964ea08b6 quay.io/k8scsi/csi-resizer:v0.1.0],SizeBytes:42623056,},ContainerImage{Names:[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1],SizeBytes:42541759,},ContainerImage{Names:[gcr.io/google-containers/debian-base@sha256:6966a0aedd7592c18ff2dd803c08bd85780ee19f5e3a2e7cf908a4cd837afcde gcr.io/google-containers/debian-base:0.4.1],SizeBytes:42323657,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nonroot@sha256:d4ede5c74517090b6686219059118ed178cf4620f5db8781b32f806bb1e7395b gcr.io/kubernetes-e2e-test-images/nonroot:1.0],SizeBytes:42321438,},ContainerImage{Names:[redis@sha256:50899ea1ceed33fa03232f3ac57578a424faa1742c1ac9c7a7bdb95cdf19b858 redis:5.0.5-alpine],SizeBytes:29331594,},ContainerImage{Names:[quay.io/k8scsi/hostpathplugin@sha256:b4826e492fc1762fceaf9726f41575ca0a4567864d3d235da874818de18039de quay.io/k8scsi/hostpathplugin:v1.2.0-rc5],SizeBytes:28761497,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7],SizeBytes:22909487,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/echoserver@sha256:e9ba514b896cdf559eef8788b66c2c3ee55f3572df617647b4b0d8b6bf81cf19 gcr.io/kubernetes-e2e-test-images/echoserver:2.2],SizeBytes:21692741,},ContainerImage{Names:[quay.io/k8scsi/mock-driver@sha256:e0eed916b7d970bad2b7d9875f9ad16932f987f0f3d91ec5d86da68b0b5cc9d1 quay.io/k8scsi/mock-driver:v2.1.0],SizeBytes:16226335,},ContainerImage{Names:[nginx@sha256:485b610fefec7ff6c463ced9623314a04ed67e3945b9c08d7e53a47f6d108dc7 nginx:1.14-alpine],SizeBytes:16032814,},ContainerImage{Names:[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0],SizeBytes:15815995,},ContainerImage{Names:[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0],SizeBytes:14967303,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/dnsutils@sha256:2abeee84efb79c14d731966e034af33bf324d3b26ca28497555511ff094b3ddd gcr.io/kubernetes-e2e-test-images/dnsutils:1.1],SizeBytes:9349974,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nonewprivs@sha256:10066e9039219449fe3c81f38fe01928f87914150768ab81b62a468e51fa7411 gcr.io/kubernetes-e2e-test-images/nonewprivs:1.0],SizeBytes:6757579,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nautilus@sha256:33a732d4c42a266912a5091598a0f07653c9134db4b8d571690d8afd509e0bfc gcr.io/kubernetes-e2e-test-images/nautilus:1.0],SizeBytes:4753501,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/kitten@sha256:bcbc4875c982ab39aa7c4f6acf4a287f604e996d9f34a3fbda8c3d1a7457d1f6 gcr.io/kubernetes-e2e-test-images/kitten:1.0],SizeBytes:4747037,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/test-webserver@sha256:7f93d6e32798ff28bc6289254d0c2867fe2c849c8e46edc50f8624734309812e gcr.io/kubernetes-e2e-test-images/test-webserver:1.0],SizeBytes:4732240,},ContainerImage{Names:[alpine@sha256:8421d9a84432575381bfabd248f1eb56f3aa21d9d7cd2511583c68c9b7511d10 alpine:3.7],SizeBytes:4206494,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 gcr.io/kubernetes-e2e-test-images/mounttest:1.0],SizeBytes:1563521,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest-user@sha256:17319ca525ee003681fccf7e8c6b1b910ff4f49b653d939ac7f9b6e7c463933d gcr.io/kubernetes-e2e-test-images/mounttest-user:1.0],SizeBytes:1450451,},ContainerImage{Names:[busybox@sha256:1303dbf110c57f3edf68d9f5a16c082ec06c4cf7604831669faf2c712260b5a0 busybox:latest],SizeBytes:1219790,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea k8s.gcr.io/pause-amd64:3.1 k8s.gcr.io/pause:3.1],SizeBytes:742472,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-provisioning-7066^76296861-0659-11ea-96d2-000d3ac2cd71],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-provisioning-7066^76296861-0659-11ea-96d2-000d3ac2cd71,DevicePath:,},},Config:nil,},} Nov 13 21:06:44.683: INFO: Logging kubelet events for node k8s-agentpool-24944505-vmss000000 Nov 13 21:06:44.740: INFO: Logging pods the kubelet thinks is on node k8s-agentpool-24944505-vmss000000 Nov 13 21:06:44.872: INFO: keyvault-flexvolume-sn47m started at 2019-11-13 20:42:59 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:44.872: INFO: Container keyvault-flexvolume ready: true, restart count 0 Nov 13 21:06:44.872: INFO: blobfuse-flexvol-installer-drhkj started at 2019-11-13 20:42:59 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:44.872: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 Nov 13 21:06:44.872: INFO: pod-secrets-6844b856-1c16-40d1-b849-d18419df8fea started at 2019-11-13 21:06:33 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:44.872: INFO: Container secret-env-test ready: false, restart count 0 Nov 13 21:06:44.872: INFO: liveness-d82b4b32-ca11-41ae-aaa2-64e320c5c211 started at 2019-11-13 21:04:50 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:44.872: INFO: Container liveness ready: true, restart count 0 Nov 13 21:06:44.872: INFO: csi-snapshotter-0 started at 2019-11-13 21:05:48 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:44.872: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 13 21:06:44.872: INFO: csi-mockplugin-attacher-0 started at 2019-11-13 21:06:43 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:44.872: INFO: Container csi-attacher ready: false, restart count 0 Nov 13 21:06:44.872: INFO: server-envvars-4b710ced-7c1b-4260-835f-246ded177c37 started at 2019-11-13 21:04:40 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:44.872: INFO: Container srv ready: true, restart count 0 Nov 13 21:06:44.872: INFO: netserver-0 started at 2019-11-13 21:04:45 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:44.872: INFO: Container webserver ready: true, restart count 0 Nov 13 21:06:44.872: INFO: rs-ldwc2 started at 2019-11-13 21:04:56 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:44.872: INFO: Container busybox ready: true, restart count 0 Nov 13 21:06:44.872: INFO: liveness-d2a47201-2d54-48f6-8da5-33b2be6ef5f4 started at 2019-11-13 21:05:20 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:44.872: INFO: Container liveness ready: true, restart count 1 Nov 13 21:06:44.872: INFO: kube-proxy-h9n4c started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:44.872: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 21:06:44.872: INFO: kubernetes-dashboard-65966766b9-lq4lm started at 2019-11-13 20:42:59 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:44.872: INFO: Container kubernetes-dashboard ready: true, restart count 0 Nov 13 21:06:44.872: INFO: pod-secrets-c83482db-7746-4e0b-8a68-0009af614dbe started at 2019-11-13 21:06:30 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:44.872: INFO: Container secret-volume-test ready: false, restart count 0 Nov 13 21:06:44.872: INFO: metrics-server-58ff8c5ddf-r2d7f started at 2019-11-13 20:42:59 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:44.872: INFO: Container metrics-server ready: true, restart count 0 Nov 13 21:06:44.872: INFO: csi-hostpath-resizer-0 started at 2019-11-13 21:05:47 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:44.872: INFO: Container csi-resizer ready: true, restart count 0 Nov 13 21:06:44.872: INFO: azure-ip-masq-agent-qksp6 started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:44.872: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 21:06:44.872: INFO: tiller-deploy-7559b6b885-qxwkm started at 2019-11-13 20:42:59 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:44.872: INFO: Container tiller ready: true, restart count 0 Nov 13 21:06:44.872: INFO: rs-mxw48 started at 2019-11-13 21:04:51 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:44.872: INFO: Container busybox ready: true, restart count 0 Nov 13 21:06:44.872: INFO: pod-with-poststart-exec-hook started at 2019-11-13 21:05:31 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:44.872: INFO: Container pod-with-poststart-exec-hook ready: false, restart count 0 Nov 13 21:06:44.872: INFO: pod-subpath-test-local-preprovisionedpv-nmd7 started at 2019-11-13 21:05:54 +0000 UTC (2+1 container statuses recorded) Nov 13 21:06:44.872: INFO: Init container init-volume-local-preprovisionedpv-nmd7 ready: true, restart count 0 Nov 13 21:06:44.872: INFO: Init container test-init-volume-local-preprovisionedpv-nmd7 ready: false, restart count 0 Nov 13 21:06:44.872: INFO: Container test-container-subpath-local-preprovisionedpv-nmd7 ready: false, restart count 0 Nov 13 21:06:44.872: INFO: netserver-0 started at 2019-11-13 21:06:24 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:44.872: INFO: Container webserver ready: false, restart count 0 Nov 13 21:06:44.872: INFO: pvc-tester-85ntb started at 2019-11-13 21:06:35 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:44.872: INFO: Container write-pod ready: false, restart count 0 Nov 13 21:06:44.872: INFO: hostexec-k8s-agentpool-24944505-vmss000000 started at 2019-11-13 21:04:48 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:44.872: INFO: Container agnhost ready: true, restart count 0 Nov 13 21:06:44.872: INFO: rs-mghwh started at 2019-11-13 21:04:52 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:44.872: INFO: Container busybox ready: true, restart count 0 Nov 13 21:06:44.872: INFO: hostexec-k8s-agentpool-24944505-vmss000000 started at 2019-11-13 21:06:09 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:44.872: INFO: Container agnhost ready: true, restart count 0 Nov 13 21:06:44.872: INFO: rs-6b4qm started at 2019-11-13 21:04:53 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:44.872: INFO: Container busybox ready: true, restart count 0 Nov 13 21:06:44.872: INFO: pod-subpath-test-csi-hostpath-dynamicpv-qnhc started at 2019-11-13 21:06:32 +0000 UTC (1+1 container statuses recorded) Nov 13 21:06:44.872: INFO: Init container init-volume-csi-hostpath-dynamicpv-qnhc ready: false, restart count 0 Nov 13 21:06:44.872: INFO: Container test-container-subpath-csi-hostpath-dynamicpv-qnhc ready: false, restart count 0 Nov 13 21:06:44.872: INFO: netserver-0 started at 2019-11-13 21:06:02 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:44.872: INFO: Container webserver ready: false, restart count 0 Nov 13 21:06:44.872: INFO: rs-8v6pt started at 2019-11-13 21:04:55 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:44.872: INFO: Container busybox ready: true, restart count 0 Nov 13 21:06:44.872: INFO: busybox-readonly-false-c0aa7944-54ef-4cae-ad4e-78a89e3be653 started at 2019-11-13 21:06:15 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:44.872: INFO: Container busybox-readonly-false-c0aa7944-54ef-4cae-ad4e-78a89e3be653 ready: false, restart count 0 Nov 13 21:06:44.872: INFO: pod-subpath-test-local-preprovisionedpv-9brd started at 2019-11-13 21:06:38 +0000 UTC (1+1 container statuses recorded) Nov 13 21:06:44.872: INFO: Init container init-volume-local-preprovisionedpv-9brd ready: false, restart count 0 Nov 13 21:06:44.872: INFO: Container test-container-subpath-local-preprovisionedpv-9brd ready: false, restart count 0 Nov 13 21:06:44.872: INFO: csi-hostpath-attacher-0 started at 2019-11-13 21:05:43 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:44.872: INFO: Container csi-attacher ready: true, restart count 0 Nov 13 21:06:44.872: INFO: alpine-nnp-nil-70d591f4-0613-4bc7-b7b1-109034edded2 started at 2019-11-13 21:06:32 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:44.872: INFO: Container alpine-nnp-nil-70d591f4-0613-4bc7-b7b1-109034edded2 ready: false, restart count 0 Nov 13 21:06:44.872: INFO: pod-subpath-test-emptydir-z5gk started at 2019-11-13 21:06:27 +0000 UTC (1+1 container statuses recorded) Nov 13 21:06:44.872: INFO: Init container init-volume-emptydir-z5gk ready: false, restart count 0 Nov 13 21:06:44.872: INFO: Container test-container-subpath-emptydir-z5gk ready: false, restart count 0 Nov 13 21:06:44.872: INFO: pod-hostip-35cbf031-c107-4d09-96f4-504504ccd49e started at 2019-11-13 21:06:36 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:44.872: INFO: Container test ready: false, restart count 0 Nov 13 21:06:44.872: INFO: csi-mockplugin-0 started at 2019-11-13 21:06:42 +0000 UTC (0+3 container statuses recorded) Nov 13 21:06:44.872: INFO: Container csi-provisioner ready: false, restart count 0 Nov 13 21:06:44.872: INFO: Container driver-registrar ready: false, restart count 0 Nov 13 21:06:44.872: INFO: Container mock ready: false, restart count 0 Nov 13 21:06:44.872: INFO: csi-hostpathplugin-0 started at 2019-11-13 21:05:45 +0000 UTC (0+3 container statuses recorded) Nov 13 21:06:44.872: INFO: Container hostpath ready: false, restart count 0 Nov 13 21:06:44.872: INFO: Container liveness-probe ready: false, restart count 0 Nov 13 21:06:44.872: INFO: Container node-driver-registrar ready: false, restart count 0 Nov 13 21:06:44.872: INFO: pod-projected-configmaps-15a8f794-a2e5-4a03-aeda-2dd454aaedbf started at 2019-11-13 21:04:47 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:44.872: INFO: Container projected-configmap-volume-test ready: false, restart count 0 Nov 13 21:06:44.872: INFO: csi-hostpath-provisioner-0 started at 2019-11-13 21:05:46 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:44.872: INFO: Container csi-provisioner ready: true, restart count 0 W1113 21:06:44.927186 92656 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 21:06:46.624: INFO: Latency metrics for node k8s-agentpool-24944505-vmss000000 Nov 13 21:06:46.624: INFO: Logging node info for node k8s-agentpool-24944505-vmss000001 Nov 13 21:06:46.678: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool-24944505-vmss000001 /api/v1/nodes/k8s-agentpool-24944505-vmss000001 da675d2e-0aad-4a8f-b94d-1311fb000563 38121 0 2019-11-13 20:42:24 +0000 UTC <nil> <nil> map[agentpool:agentpool beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool-24944505-vmss000001 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: storageprofile:managed storagetier:Premium_LRS] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-351":"k8s-agentpool-24944505-vmss000001","csi-hostpath-ephemeral-5584":"k8s-agentpool-24944505-vmss000001","csi-hostpath-provisioning-1638":"k8s-agentpool-24944505-vmss000001","csi-hostpath-provisioning-225":"k8s-agentpool-24944505-vmss000001","csi-hostpath-provisioning-7099":"k8s-agentpool-24944505-vmss000001","csi-hostpath-volume-6291":"k8s-agentpool-24944505-vmss000001","csi-hostpath-volume-expand-8458":"k8s-agentpool-24944505-vmss000001","csi-mock-csi-mock-volumes-4469":"csi-mock-csi-mock-volumes-4469","csi-mock-csi-mock-volumes-4959":"csi-mock-csi-mock-volumes-4959","csi-mock-csi-mock-volumes-597":"csi-mock-csi-mock-volumes-597","csi-mock-csi-mock-volumes-8812":"csi-mock-csi-mock-volumes-8812","csi-mock-csi-mock-volumes-9088":"csi-mock-csi-mock-volumes-9088"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-24944505-vmss/virtualMachines/1,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16797569024 0} {<nil>} 16403876Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16011137024 0} {<nil>} 15635876Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 21:06:31 +0000 UTC,LastTransitionTime:2019-11-13 20:42:24 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 21:06:31 +0000 UTC,LastTransitionTime:2019-11-13 20:42:24 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 21:06:31 +0000 UTC,LastTransitionTime:2019-11-13 20:42:24 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 21:06:31 +0000 UTC,LastTransitionTime:2019-11-13 20:42:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.248.0.5,},NodeAddress{Type:Hostname,Address:k8s-agentpool-24944505-vmss000001,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:6ed41b2c4b884fccaf94748594531e06,SystemUUID:9CE5C2E3-7C34-C64A-9734-2477E19937B7,BootID:306ac3f6-a23b-423e-83c9-e3d5f3938f6c,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[quay.io/kubernetes_incubator/nfs-provisioner@sha256:df762117e3c891f2d2ddff46ecb0776ba1f9f3c44cfd7739b0683bcd7a7954a8 quay.io/kubernetes_incubator/nfs-provisioner:v2.2.2],SizeBytes:391772778,},ContainerImage{Names:[gluster/glusterdynamic-provisioner@sha256:90067cb05a7d217651e84576935934fd9dcff8e6bcdcbaa416bbf36fcd09dbd1 gluster/glusterdynamic-provisioner:v1.0],SizeBytes:373281573,},ContainerImage{Names:[gcr.io/google-samples/gb-frontend@sha256:35cb427341429fac3df10ff74600ea73e8ec0754d78f9ce89e0b4f3d70d53ba6 gcr.io/google-samples/gb-frontend:v6],SizeBytes:373099368,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/volume/nfs@sha256:c2ad734346f608a5f7d69cfded93c4e8094069320657bd372d12ba21dea3ea71 gcr.io/kubernetes-e2e-test-images/volume/nfs:1.0],SizeBytes:225358913,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/jessie-dnsutils@sha256:ad583e33cb284f7ef046673809b146ec4053cda19b54a85d2b180a86169715eb gcr.io/kubernetes-e2e-test-images/jessie-dnsutils:1.0],SizeBytes:195659796,},ContainerImage{Names:[httpd@sha256:addd70e4ee83f3bc9a4c1c7c41e37927ba47faf639312fc936df3afad7926f5a httpd:2.4.39-alpine],SizeBytes:126894770,},ContainerImage{Names:[httpd@sha256:eb8ccf084cf3e80eece1add239effefd171eb39adbc154d33c14260d905d4060 httpd:2.4.38-alpine],SizeBytes:123781643,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727 gcr.io/kubernetes-e2e-test-images/agnhost:2.6],SizeBytes:57345321,},ContainerImage{Names:[quay.io/k8scsi/csi-provisioner@sha256:0efcb424f1dde9b9fb11a1a14f2e48ab47e1c3f08bc3a929990dcfcb1f7ab34f quay.io/k8scsi/csi-provisioner:v1.4.0-rc1],SizeBytes:54431016,},ContainerImage{Names:[quay.io/k8scsi/csi-snapshotter@sha256:e3d3e742e32d00488fdb401045b9b1d033d7ca0ab6e760f77b24750fc95e5f70 quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1],SizeBytes:51703561,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0],SizeBytes:46226754,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:0aba670b4d9d6b2e720bbf575d733156c676b693ca26501235444490300db838 quay.io/k8scsi/csi-attacher:v1.1.0],SizeBytes:42839085,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:7d46fb6eb8b890dc546029d1565d502b4a1d974d33625c6ee2bc7991b77fc1a1 quay.io/k8scsi/csi-resizer:v0.2.0],SizeBytes:42817100,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:f315c9042e56def3c05c6b04fe79ec9da6d39ddc557ca365a76cf35964ea08b6 quay.io/k8scsi/csi-resizer:v0.1.0],SizeBytes:42623056,},ContainerImage{Names:[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0],SizeBytes:42488424,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nonroot@sha256:d4ede5c74517090b6686219059118ed178cf4620f5db8781b32f806bb1e7395b gcr.io/kubernetes-e2e-test-images/nonroot:1.0],SizeBytes:42321438,},ContainerImage{Names:[redis@sha256:50899ea1ceed33fa03232f3ac57578a424faa1742c1ac9c7a7bdb95cdf19b858 redis:5.0.5-alpine],SizeBytes:29331594,},ContainerImage{Names:[quay.io/k8scsi/hostpathplugin@sha256:b4826e492fc1762fceaf9726f41575ca0a4567864d3d235da874818de18039de quay.io/k8scsi/hostpathplugin:v1.2.0-rc5],SizeBytes:28761497,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7],SizeBytes:22909487,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/echoserver@sha256:e9ba514b896cdf559eef8788b66c2c3ee55f3572df617647b4b0d8b6bf81cf19 gcr.io/kubernetes-e2e-test-images/echoserver:2.2],SizeBytes:21692741,},ContainerImage{Names:[quay.io/k8scsi/mock-driver@sha256:e0eed916b7d970bad2b7d9875f9ad16932f987f0f3d91ec5d86da68b0b5cc9d1 quay.io/k8scsi/mock-driver:v2.1.0],SizeBytes:16226335,},ContainerImage{Names:[nginx@sha256:485b610fefec7ff6c463ced9623314a04ed67e3945b9c08d7e53a47f6d108dc7 nginx:1.14-alpine],SizeBytes:16032814,},ContainerImage{Names:[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0],SizeBytes:15815995,},ContainerImage{Names:[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0],SizeBytes:14967303,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/dnsutils@sha256:2abeee84efb79c14d731966e034af33bf324d3b26ca28497555511ff094b3ddd gcr.io/kubernetes-e2e-test-images/dnsutils:1.1],SizeBytes:9349974,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nautilus@sha256:33a732d4c42a266912a5091598a0f07653c9134db4b8d571690d8afd509e0bfc gcr.io/kubernetes-e2e-test-images/nautilus:1.0],SizeBytes:4753501,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/kitten@sha256:bcbc4875c982ab39aa7c4f6acf4a287f604e996d9f34a3fbda8c3d1a7457d1f6 gcr.io/kubernetes-e2e-test-images/kitten:1.0],SizeBytes:4747037,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/test-webserver@sha256:7f93d6e32798ff28bc6289254d0c2867fe2c849c8e46edc50f8624734309812e gcr.io/kubernetes-e2e-test-images/test-webserver:1.0],SizeBytes:4732240,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:4206620,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 gcr.io/kubernetes-e2e-test-images/mounttest:1.0],SizeBytes:1563521,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea k8s.gcr.io/pause-amd64:3.1 k8s.gcr.io/pause:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 21:06:46.678: INFO: Logging kubelet events for node k8s-agentpool-24944505-vmss000001 Nov 13 21:06:46.735: INFO: Logging pods the kubelet thinks is on node k8s-agentpool-24944505-vmss000001 Nov 13 21:06:46.896: INFO: nfs-server started at 2019-11-13 21:05:55 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:46.897: INFO: Container nfs-server ready: true, restart count 0 Nov 13 21:06:46.897: INFO: csi-mockplugin-resizer-0 started at 2019-11-13 21:02:56 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:46.897: INFO: Container csi-resizer ready: true, restart count 0 Nov 13 21:06:46.897: INFO: client-containers-b0523daf-64b7-4db3-a9c1-e67d73c0b9c6 started at 2019-11-13 21:06:44 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:46.897: INFO: Container test-container ready: false, restart count 0 Nov 13 21:06:46.897: INFO: pod-handle-http-request started at 2019-11-13 21:04:25 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:46.897: INFO: Container pod-handle-http-request ready: true, restart count 0 Nov 13 21:06:46.897: INFO: pod-subpath-test-hostpath-hvg9 started at 2019-11-13 21:05:05 +0000 UTC (0+2 container statuses recorded) Nov 13 21:06:46.897: INFO: Container test-container-subpath-hostpath-hvg9 ready: false, restart count 0 Nov 13 21:06:46.897: INFO: Container test-container-volume-hostpath-hvg9 ready: true, restart count 0 Nov 13 21:06:46.897: INFO: csi-snapshotter-0 started at 2019-11-13 21:06:31 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:46.897: INFO: Container csi-snapshotter ready: false, restart count 0 Nov 13 21:06:46.897: INFO: pod-subpath-test-local-preprovisionedpv-dqs2 started at 2019-11-13 21:06:35 +0000 UTC (1+1 container statuses recorded) Nov 13 21:06:46.897: INFO: Init container init-volume-local-preprovisionedpv-dqs2 ready: false, restart count 0 Nov 13 21:06:46.897: INFO: Container test-container-subpath-local-preprovisionedpv-dqs2 ready: false, restart count 0 Nov 13 21:06:46.897: INFO: coredns-87f5d796-jd8bc started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:46.897: INFO: Container coredns ready: true, restart count 0 Nov 13 21:06:46.897: INFO: rs-wftlk started at 2019-11-13 21:04:52 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:46.897: INFO: Container busybox ready: true, restart count 0 Nov 13 21:06:46.897: INFO: pod-ready started at 2019-11-13 21:05:41 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:46.897: INFO: Container pod-readiness-gate ready: true, restart count 0 Nov 13 21:06:46.897: INFO: csi-mockplugin-attacher-0 started at 2019-11-13 21:02:55 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:46.897: INFO: Container csi-attacher ready: true, restart count 0 Nov 13 21:06:46.897: INFO: netserver-1 started at 2019-11-13 21:06:02 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:46.897: INFO: Container webserver ready: true, restart count 0 Nov 13 21:06:46.897: INFO: keyvault-flexvolume-7phpr started at 2019-11-13 20:42:58 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:46.897: INFO: Container keyvault-flexvolume ready: true, restart count 0 Nov 13 21:06:46.897: INFO: csi-hostpath-provisioner-0 started at 2019-11-13 21:06:28 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:46.897: INFO: Container csi-provisioner ready: false, restart count 0 Nov 13 21:06:46.897: INFO: csi-mockplugin-0 started at 2019-11-13 21:04:08 +0000 UTC (0+3 container statuses recorded) Nov 13 21:06:46.897: INFO: Container csi-provisioner ready: true, restart count 0 Nov 13 21:06:46.897: INFO: Container driver-registrar ready: true, restart count 0 Nov 13 21:06:46.897: INFO: Container mock ready: true, restart count 0 Nov 13 21:06:46.897: INFO: rs-cnqj7 started at 2019-11-13 21:04:50 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:46.897: INFO: Container busybox ready: true, restart count 0 Nov 13 21:06:46.897: INFO: blobfuse-flexvol-installer-cpm6h started at 2019-11-13 20:42:58 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:46.897: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 Nov 13 21:06:46.897: INFO: azure-ip-masq-agent-wwpg7 started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:46.897: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 21:06:46.897: INFO: rs-sffc7 started at 2019-11-13 21:04:53 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:46.897: INFO: Container busybox ready: true, restart count 0 Nov 13 21:06:46.897: INFO: kube-proxy-7bpf7 started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:46.897: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 21:06:46.897: INFO: netserver-1 started at 2019-11-13 21:06:29 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:46.897: INFO: Container webserver ready: false, restart count 0 Nov 13 21:06:46.897: INFO: rs-n9tx2 started at 2019-11-13 21:04:54 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:46.897: INFO: Container busybox ready: true, restart count 0 Nov 13 21:06:46.897: INFO: hostexec-k8s-agentpool-24944505-vmss000001 started at 2019-11-13 21:05:42 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:46.897: INFO: Container agnhost ready: true, restart count 0 Nov 13 21:06:46.897: INFO: csi-mockplugin-0 started at 2019-11-13 21:02:54 +0000 UTC (0+3 container statuses recorded) Nov 13 21:06:46.897: INFO: Container csi-provisioner ready: true, restart count 0 Nov 13 21:06:46.897: INFO: Container driver-registrar ready: true, restart count 0 Nov 13 21:06:46.897: INFO: Container mock ready: true, restart count 0 Nov 13 21:06:46.897: INFO: local-injector started at 2019-11-13 21:06:04 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:46.897: INFO: Container local-injector ready: true, restart count 0 Nov 13 21:06:46.897: INFO: csi-mockplugin-attacher-0 started at 2019-11-13 21:04:09 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:46.897: INFO: Container csi-attacher ready: true, restart count 0 Nov 13 21:06:46.897: INFO: sample-webhook-deployment-86d95b659d-g98sv started at 2019-11-13 21:06:17 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:46.897: INFO: Container sample-webhook ready: false, restart count 0 Nov 13 21:06:46.897: INFO: csi-hostpath-resizer-0 started at 2019-11-13 21:06:30 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:46.897: INFO: Container csi-resizer ready: false, restart count 0 Nov 13 21:06:46.897: INFO: rs-smhmv started at 2019-11-13 21:04:51 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:46.897: INFO: Container busybox ready: true, restart count 0 Nov 13 21:06:46.897: INFO: hostexec-k8s-agentpool-24944505-vmss000001 started at 2019-11-13 21:05:09 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:46.897: INFO: Container agnhost ready: true, restart count 0 Nov 13 21:06:46.897: INFO: test-webserver-063a973c-59e0-4e62-abe0-ed65b5b8a89d started at 2019-11-13 21:02:51 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:46.897: INFO: Container test-webserver ready: true, restart count 0 Nov 13 21:06:46.897: INFO: csi-hostpath-attacher-0 started at 2019-11-13 21:06:25 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:46.897: INFO: Container csi-attacher ready: false, restart count 0 Nov 13 21:06:46.897: INFO: csi-hostpathplugin-0 started at 2019-11-13 21:06:27 +0000 UTC (0+3 container statuses recorded) Nov 13 21:06:46.897: INFO: Container hostpath ready: false, restart count 0 Nov 13 21:06:46.897: INFO: Container liveness-probe ready: false, restart count 0 Nov 13 21:06:46.897: INFO: Container node-driver-registrar ready: false, restart count 0 W1113 21:06:46.951317 92656 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 21:06:47.084: INFO: Latency metrics for node k8s-agentpool-24944505-vmss000001 Nov 13 21:06:47.084: INFO: Logging node info for node k8s-master-24944505-vmss000000 Nov 13 21:06:47.137: INFO: Node Info: &Node{ObjectMeta:{k8s-master-24944505-vmss000000 /api/v1/nodes/k8s-master-24944505-vmss000000 048fbae2-b97f-43a2-b471-398180136f12 37485 0 2019-11-13 20:42:13 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-24944505-vmss000000 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-24944505-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284887552 0} {<nil>} 7114148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6498455552 0} {<nil>} 6346148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 21:06:07 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 21:06:07 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 21:06:07 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 21:06:07 +0000 UTC,LastTransitionTime:2019-11-13 20:42:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.4,},NodeAddress{Type:Hostname,Address:k8s-master-24944505-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:7bd754c0dc8647518c3043bcae3a76cc,SystemUUID:FA5CFE99-8148-6C46-B4E6-5EA2A5DE9D2A,BootID:74f7b0a2-e85c-438e-a642-c5b32900bb28,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:3237a53b135d8692ade82eca845fd0259ecbfcbe05fad35400e7d5c2018c4abd k8sprow.azurecr.io/azure-cloud-controller-manager:1194710255657816064-e688ecb0],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 21:06:47.137: INFO: Logging kubelet events for node k8s-master-24944505-vmss000000 Nov 13 21:06:47.195: INFO: Logging pods the kubelet thinks is on node k8s-master-24944505-vmss000000 Nov 13 21:06:47.270: INFO: cloud-controller-manager-k8s-master-24944505-vmss000000 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:47.270: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 13 21:06:47.270: INFO: kube-addon-manager-k8s-master-24944505-vmss000000 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:47.270: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 13 21:06:47.270: INFO: kube-apiserver-k8s-master-24944505-vmss000000 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:47.270: INFO: Container kube-apiserver ready: true, restart count 0 Nov 13 21:06:47.270: INFO: kube-controller-manager-k8s-master-24944505-vmss000000 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:47.270: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 13 21:06:47.270: INFO: kube-scheduler-k8s-master-24944505-vmss000000 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:47.270: INFO: Container kube-scheduler ready: true, restart count 0 Nov 13 21:06:47.270: INFO: azure-ip-masq-agent-6pnst started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:47.270: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 21:06:47.270: INFO: kube-proxy-d9jjh started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:47.270: INFO: Container kube-proxy ready: true, restart count 0 W1113 21:06:47.326321 92656 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 21:06:47.443: INFO: Latency metrics for node k8s-master-24944505-vmss000000 Nov 13 21:06:47.443: INFO: Logging node info for node k8s-master-24944505-vmss000001 Nov 13 21:06:47.495: INFO: Node Info: &Node{ObjectMeta:{k8s-master-24944505-vmss000001 /api/v1/nodes/k8s-master-24944505-vmss000001 a2619878-b119-44f9-b89f-4073bebbb56c 37319 0 2019-11-13 20:42:40 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-24944505-vmss000001 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.6.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-24944505-vmss/virtualMachines/1,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.6.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284883456 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6498451456 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:43:08 +0000 UTC,LastTransitionTime:2019-11-13 20:43:08 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 21:06:03 +0000 UTC,LastTransitionTime:2019-11-13 20:42:34 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 21:06:03 +0000 UTC,LastTransitionTime:2019-11-13 20:42:34 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 21:06:03 +0000 UTC,LastTransitionTime:2019-11-13 20:42:34 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 21:06:03 +0000 UTC,LastTransitionTime:2019-11-13 20:42:40 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.5,},NodeAddress{Type:Hostname,Address:k8s-master-24944505-vmss000001,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:c866a7c1ed64481990416e24387dd317,SystemUUID:EBE64557-2AC8-5740-9FFE-AB9F12412FBC,BootID:3dbb0e0e-0479-4821-bc3f-a8269d250e4c,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:3237a53b135d8692ade82eca845fd0259ecbfcbe05fad35400e7d5c2018c4abd k8sprow.azurecr.io/azure-cloud-controller-manager:1194710255657816064-e688ecb0],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 21:06:47.495: INFO: Logging kubelet events for node k8s-master-24944505-vmss000001 Nov 13 21:06:47.556: INFO: Logging pods the kubelet thinks is on node k8s-master-24944505-vmss000001 Nov 13 21:06:47.633: INFO: kube-scheduler-k8s-master-24944505-vmss000001 started at 2019-11-13 20:42:28 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:47.633: INFO: Container kube-scheduler ready: true, restart count 0 Nov 13 21:06:47.633: INFO: cloud-controller-manager-k8s-master-24944505-vmss000001 started at 2019-11-13 20:42:28 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:47.633: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 13 21:06:47.633: INFO: kube-addon-manager-k8s-master-24944505-vmss000001 started at 2019-11-13 20:42:28 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:47.633: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 13 21:06:47.633: INFO: kube-apiserver-k8s-master-24944505-vmss000001 started at 2019-11-13 20:42:28 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:47.633: INFO: Container kube-apiserver ready: true, restart count 0 Nov 13 21:06:47.633: INFO: kube-controller-manager-k8s-master-24944505-vmss000001 started at 2019-11-13 20:42:28 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:47.633: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 13 21:06:47.633: INFO: kube-proxy-w4bz5 started at 2019-11-13 20:42:41 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:47.633: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 21:06:47.633: INFO: azure-ip-masq-agent-bs52d started at 2019-11-13 20:42:41 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:47.633: INFO: Container azure-ip-masq-agent ready: true, restart count 0 W1113 21:06:47.686773 92656 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 21:06:47.806: INFO: Latency metrics for node k8s-master-24944505-vmss000001 Nov 13 21:06:47.807: INFO: Logging node info for node k8s-master-24944505-vmss000002 Nov 13 21:06:47.859: INFO: Node Info: &Node{ObjectMeta:{k8s-master-24944505-vmss000002 /api/v1/nodes/k8s-master-24944505-vmss000002 f3b05267-a009-4dba-9f97-4f772e48707e 37469 0 2019-11-13 20:42:13 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-24944505-vmss000002 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.4.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-24944505-vmss/virtualMachines/2,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284887552 0} {<nil>} 7114148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6498455552 0} {<nil>} 6346148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 21:06:07 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 21:06:07 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 21:06:07 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 21:06:07 +0000 UTC,LastTransitionTime:2019-11-13 20:42:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.6,},NodeAddress{Type:Hostname,Address:k8s-master-24944505-vmss000002,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:fc1777f37ced4b9c97acb9b30e276c4f,SystemUUID:E9DF19DC-D02C-714C-A7D6-0BA6CE959C55,BootID:622958a8-d45d-4edf-9266-4dbb759e2359,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:3237a53b135d8692ade82eca845fd0259ecbfcbe05fad35400e7d5c2018c4abd k8sprow.azurecr.io/azure-cloud-controller-manager:1194710255657816064-e688ecb0],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 21:06:47.860: INFO: Logging kubelet events for node k8s-master-24944505-vmss000002 Nov 13 21:06:47.916: INFO: Logging pods the kubelet thinks is on node k8s-master-24944505-vmss000002 Nov 13 21:06:48.026: INFO: kube-proxy-xwz78 started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:48.026: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 21:06:48.026: INFO: kube-addon-manager-k8s-master-24944505-vmss000002 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:48.026: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 13 21:06:48.026: INFO: kube-apiserver-k8s-master-24944505-vmss000002 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:48.026: INFO: Container kube-apiserver ready: true, restart count 0 Nov 13 21:06:48.026: INFO: kube-controller-manager-k8s-master-24944505-vmss000002 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:48.026: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 13 21:06:48.026: INFO: kube-scheduler-k8s-master-24944505-vmss000002 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:48.026: INFO: Container kube-scheduler ready: true, restart count 0 Nov 13 21:06:48.027: INFO: cloud-controller-manager-k8s-master-24944505-vmss000002 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:48.027: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 13 21:06:48.027: INFO: azure-ip-masq-agent-6skf2 started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:48.027: INFO: Container azure-ip-masq-agent ready: true, restart count 0 W1113 21:06:48.081467 92656 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 21:06:48.197: INFO: Latency metrics for node k8s-master-24944505-vmss000002 Nov 13 21:06:48.197: INFO: Logging node info for node k8s-master-24944505-vmss000003 Nov 13 21:06:48.249: INFO: Node Info: &Node{ObjectMeta:{k8s-master-24944505-vmss000003 /api/v1/nodes/k8s-master-24944505-vmss000003 888cc19e-c62d-4945-a425-97d14e38a0b8 37210 0 2019-11-13 20:42:35 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-24944505-vmss000003 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.5.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-24944505-vmss/virtualMachines/3,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.5.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284887552 0} {<nil>} 7114148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6498455552 0} {<nil>} 6346148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 21:05:59 +0000 UTC,LastTransitionTime:2019-11-13 20:42:29 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 21:05:59 +0000 UTC,LastTransitionTime:2019-11-13 20:42:29 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 21:05:59 +0000 UTC,LastTransitionTime:2019-11-13 20:42:29 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 21:05:59 +0000 UTC,LastTransitionTime:2019-11-13 20:42:35 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.7,},NodeAddress{Type:Hostname,Address:k8s-master-24944505-vmss000003,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:2da5fe5a442b4e178a2b62ba91366e4b,SystemUUID:D2D1E816-D70E-3D40-B959-B00A2B76A83A,BootID:c5929aa1-3dac-45d5-bb2c-174c73c7300e,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:3237a53b135d8692ade82eca845fd0259ecbfcbe05fad35400e7d5c2018c4abd k8sprow.azurecr.io/azure-cloud-controller-manager:1194710255657816064-e688ecb0],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 21:06:48.249: INFO: Logging kubelet events for node k8s-master-24944505-vmss000003 Nov 13 21:06:48.305: INFO: Logging pods the kubelet thinks is on node k8s-master-24944505-vmss000003 Nov 13 21:06:48.383: INFO: kube-apiserver-k8s-master-24944505-vmss000003 started at 2019-11-13 20:42:23 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:48.383: INFO: Container kube-apiserver ready: true, restart count 0 Nov 13 21:06:48.383: INFO: kube-controller-manager-k8s-master-24944505-vmss000003 started at 2019-11-13 20:42:23 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:48.383: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 13 21:06:48.383: INFO: kube-scheduler-k8s-master-24944505-vmss000003 started at 2019-11-13 20:42:23 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:48.383: INFO: Container kube-scheduler ready: true, restart count 0 Nov 13 21:06:48.383: INFO: cloud-controller-manager-k8s-master-24944505-vmss000003 started at 2019-11-13 20:42:44 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:48.383: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 13 21:06:48.383: INFO: azure-ip-masq-agent-nlqss started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:48.383: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 21:06:48.383: INFO: kube-proxy-kgh5h started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:48.383: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 21:06:48.383: INFO: kube-addon-manager-k8s-master-24944505-vmss000003 started at 2019-11-13 20:42:22 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:48.383: INFO: Container kube-addon-manager ready: true, restart count 0 W1113 21:06:48.436063 92656 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 21:06:48.554: INFO: Latency metrics for node k8s-master-24944505-vmss000003 Nov 13 21:06:48.554: INFO: Logging node info for node k8s-master-24944505-vmss000004 Nov 13 21:06:48.607: INFO: Node Info: &Node{ObjectMeta:{k8s-master-24944505-vmss000004 /api/v1/nodes/k8s-master-24944505-vmss000004 a26d844f-0612-406b-9a65-7e2ac952a2d5 37488 0 2019-11-13 20:42:14 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-24944505-vmss000004 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-24944505-vmss/virtualMachines/4,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284887552 0} {<nil>} 7114148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6498455552 0} {<nil>} 6346148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 21:06:07 +0000 UTC,LastTransitionTime:2019-11-13 20:42:09 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 21:06:07 +0000 UTC,LastTransitionTime:2019-11-13 20:42:09 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 21:06:07 +0000 UTC,LastTransitionTime:2019-11-13 20:42:09 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 21:06:07 +0000 UTC,LastTransitionTime:2019-11-13 20:42:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.8,},NodeAddress{Type:Hostname,Address:k8s-master-24944505-vmss000004,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:2208959aca1045d9b48fce8d784c1e41,SystemUUID:E253DB3A-3A4D-5648-8169-B2E2976CFB4B,BootID:09bc6101-2da9-4c8e-8274-43db8c5fe1cf,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:3237a53b135d8692ade82eca845fd0259ecbfcbe05fad35400e7d5c2018c4abd k8sprow.azurecr.io/azure-cloud-controller-manager:1194710255657816064-e688ecb0],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 21:06:48.608: INFO: Logging kubelet events for node k8s-master-24944505-vmss000004 Nov 13 21:06:48.664: INFO: Logging pods the kubelet thinks is on node k8s-master-24944505-vmss000004 Nov 13 21:06:48.739: INFO: azure-ip-masq-agent-xnw2h started at 2019-11-13 20:42:38 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:48.739: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 21:06:48.739: INFO: kube-proxy-fld6p started at 2019-11-13 20:42:38 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:48.739: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 21:06:48.739: INFO: cloud-controller-manager-k8s-master-24944505-vmss000004 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:48.739: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 13 21:06:48.739: INFO: kube-addon-manager-k8s-master-24944505-vmss000004 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:48.739: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 13 21:06:48.739: INFO: kube-apiserver-k8s-master-24944505-vmss000004 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:48.739: INFO: Container kube-apiserver ready: true, restart count 0 Nov 13 21:06:48.739: INFO: kube-controller-manager-k8s-master-24944505-vmss000004 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:48.739: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 13 21:06:48.739: INFO: kube-scheduler-k8s-master-24944505-vmss000004 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:06:48.739: INFO: Container kube-scheduler ready: true, restart count 0 W1113 21:06:48.792746 92656 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 21:06:48.910: INFO: Latency metrics for node k8s-master-24944505-vmss000004 Nov 13 21:06:48.910: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "pods-7301" for this suite. Nov 13 21:08:59.127: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered Nov 13 21:09:00.881: INFO: namespace pods-7301 deletion completed in 2m11.916903789s
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[k8s\.io\]\sPrivilegedPod\s\[NodeConformance\]\sshould\senable\sprivileged\scommands\s\[LinuxOnly\]$'
test/e2e/framework/framework.go:152 Nov 13 21:32:58.199: Couldn't delete ns: "e2e-privileged-pod-805": namespace e2e-privileged-pod-805 was not deleted with limit: timed out waiting for the condition, pods remaining: 1 (&errors.errorString{s:"namespace e2e-privileged-pod-805 was not deleted with limit: timed out waiting for the condition, pods remaining: 1"}) test/e2e/framework/framework.go:336from junit_08.xml
[BeforeEach] [k8s.io] PrivilegedPod [NodeConformance] test/e2e/framework/framework.go:151 �[1mSTEP�[0m: Creating a kubernetes client Nov 13 21:20:04.018: INFO: >>> kubeConfig: /workspace/aks880331199/kubeconfig/kubeconfig.westus2.json �[1mSTEP�[0m: Building a namespace api object, basename e2e-privileged-pod �[1mSTEP�[0m: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in e2e-privileged-pod-805 �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace [It] should enable privileged commands [LinuxOnly] test/e2e/common/privileged.go:48 �[1mSTEP�[0m: Creating a pod with a privileged container �[1mSTEP�[0m: Executing in the privileged container Nov 13 21:22:52.670: INFO: ExecWithOptions {Command:[ip link add dummy1 type dummy] Namespace:e2e-privileged-pod-805 PodName:privileged-pod ContainerName:privileged-container Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false} Nov 13 21:22:52.670: INFO: >>> kubeConfig: /workspace/aks880331199/kubeconfig/kubeconfig.westus2.json Nov 13 21:22:54.612: INFO: ExecWithOptions {Command:[ip link del dummy1] Namespace:e2e-privileged-pod-805 PodName:privileged-pod ContainerName:privileged-container Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false} Nov 13 21:22:54.612: INFO: >>> kubeConfig: /workspace/aks880331199/kubeconfig/kubeconfig.westus2.json �[1mSTEP�[0m: Executing in the non-privileged container Nov 13 21:22:55.435: INFO: ExecWithOptions {Command:[ip link add dummy1 type dummy] Namespace:e2e-privileged-pod-805 PodName:privileged-pod ContainerName:not-privileged-container Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false} Nov 13 21:22:55.435: INFO: >>> kubeConfig: /workspace/aks880331199/kubeconfig/kubeconfig.westus2.json [AfterEach] [k8s.io] PrivilegedPod [NodeConformance] test/e2e/framework/framework.go:152 Nov 13 21:22:55.990: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "e2e-privileged-pod-805" for this suite. Nov 13 21:32:56.309: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered Nov 13 21:32:58.042: INFO: namespace: e2e-privileged-pod-805, resource: pods, items remaining: 1 Nov 13 21:32:58.094: INFO: namespace: e2e-privileged-pod-805, DeletionTimetamp: 2019-11-13 21:22:56 +0000 UTC, Finalizers: [kubernetes], Phase: Terminating Nov 13 21:32:58.147: INFO: namespace: e2e-privileged-pod-805, total namespaces: 14, active: 7, terminating: 7 Nov 13 21:32:58.199: INFO: POD NODE PHASE GRACE CONDITIONS Nov 13 21:32:58.199: INFO: privileged-pod k8s-agentpool-24944505-vmss000001 Running 30s [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:20:06 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:22:51 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:22:51 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:20:04 +0000 UTC }] Nov 13 21:32:58.199: INFO: Nov 13 21:32:58.199: INFO: Couldn't delete ns: "e2e-privileged-pod-805": namespace e2e-privileged-pod-805 was not deleted with limit: timed out waiting for the condition, pods remaining: 1 (&errors.errorString{s:"namespace e2e-privileged-pod-805 was not deleted with limit: timed out waiting for the condition, pods remaining: 1"})
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[k8s\.io\]\s\[sig\-node\]\sPods\sExtended\s\[k8s\.io\]\sDelete\sGrace\sPeriod\sshould\sbe\ssubmitted\sand\sremoved\s\[Conformance\]$'
test/e2e/framework/framework.go:698 Nov 13 21:18:31.444: kubelet never observed the termination notice Unexpected error: <*errors.errorString | 0xc0000a10a0>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred test/e2e/node/pods.go:163from junit_28.xml
[BeforeEach] [k8s.io] [sig-node] Pods Extended test/e2e/framework/framework.go:151 �[1mSTEP�[0m: Creating a kubernetes client Nov 13 21:16:59.867: INFO: >>> kubeConfig: /workspace/aks880331199/kubeconfig/kubeconfig.westus2.json �[1mSTEP�[0m: Building a namespace api object, basename pods �[1mSTEP�[0m: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in pods-6662 �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace [BeforeEach] [k8s.io] Delete Grace Period test/e2e/node/pods.go:47 [It] should be submitted and removed [Conformance] test/e2e/framework/framework.go:698 �[1mSTEP�[0m: creating the pod �[1mSTEP�[0m: setting up selector �[1mSTEP�[0m: submitting the pod to kubernetes �[1mSTEP�[0m: verifying the pod is in kubernetes Nov 13 21:18:00.657: INFO: Asynchronously running '/usr/local/bin/kubectl kubectl --server=https://kubetest-74ff5bf7-0652-11ea-bade-36185e316941.westus2.cloudapp.azure.com --kubeconfig=/workspace/aks880331199/kubeconfig/kubeconfig.westus2.json proxy -p 0' �[1mSTEP�[0m: deleting the pod gracefully �[1mSTEP�[0m: verifying the kubelet observed the termination notice Nov 13 21:18:31.444: FAIL: kubelet never observed the termination notice Unexpected error: <*errors.errorString | 0xc0000a10a0>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred [AfterEach] [k8s.io] [sig-node] Pods Extended test/e2e/framework/framework.go:152 �[1mSTEP�[0m: Collecting events from namespace "pods-6662". �[1mSTEP�[0m: Found 6 events. Nov 13 21:18:31.505: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for pod-submit-remove-b5ae053d-573d-4159-a262-23a1fd285799: {default-scheduler } Scheduled: Successfully assigned pods-6662/pod-submit-remove-b5ae053d-573d-4159-a262-23a1fd285799 to k8s-agentpool-24944505-vmss000000 Nov 13 21:18:31.505: INFO: At 2019-11-13 21:17:24 +0000 UTC - event for pod-submit-remove-b5ae053d-573d-4159-a262-23a1fd285799: {kubelet k8s-agentpool-24944505-vmss000000} Pulling: Pulling image "docker.io/library/nginx:1.14-alpine" Nov 13 21:18:31.505: INFO: At 2019-11-13 21:17:35 +0000 UTC - event for pod-submit-remove-b5ae053d-573d-4159-a262-23a1fd285799: {kubelet k8s-agentpool-24944505-vmss000000} Pulled: Successfully pulled image "docker.io/library/nginx:1.14-alpine" Nov 13 21:18:31.505: INFO: At 2019-11-13 21:17:42 +0000 UTC - event for pod-submit-remove-b5ae053d-573d-4159-a262-23a1fd285799: {kubelet k8s-agentpool-24944505-vmss000000} Created: Created container nginx Nov 13 21:18:31.505: INFO: At 2019-11-13 21:17:48 +0000 UTC - event for pod-submit-remove-b5ae053d-573d-4159-a262-23a1fd285799: {kubelet k8s-agentpool-24944505-vmss000000} Started: Started container nginx Nov 13 21:18:31.505: INFO: At 2019-11-13 21:18:12 +0000 UTC - event for pod-submit-remove-b5ae053d-573d-4159-a262-23a1fd285799: {kubelet k8s-agentpool-24944505-vmss000000} Killing: Stopping container nginx Nov 13 21:18:31.555: INFO: POD NODE PHASE GRACE CONDITIONS Nov 13 21:18:31.555: INFO: pod-submit-remove-b5ae053d-573d-4159-a262-23a1fd285799 k8s-agentpool-24944505-vmss000000 Running 30s [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:17:03 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:18:22 +0000 UTC ContainersNotReady containers with unready status: [nginx]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:18:22 +0000 UTC ContainersNotReady containers with unready status: [nginx]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:17:00 +0000 UTC }] Nov 13 21:18:31.555: INFO: Nov 13 21:18:31.660: INFO: Logging node info for node k8s-agentpool-24944505-vmss000000 Nov 13 21:18:31.712: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool-24944505-vmss000000 /api/v1/nodes/k8s-agentpool-24944505-vmss000000 5bc7cf9e-80da-4e26-a2c2-20757e870d68 53979 0 2019-11-13 20:42:13 +0000 UTC <nil> <nil> map[agentpool:agentpool beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool-24944505-vmss000000 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: storageprofile:managed storagetier:Premium_LRS] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-1137":"k8s-agentpool-24944505-vmss000000","csi-hostpath-ephemeral-9135":"k8s-agentpool-24944505-vmss000000","csi-hostpath-provisioning-2585":"k8s-agentpool-24944505-vmss000000","csi-hostpath-provisioning-3303":"k8s-agentpool-24944505-vmss000000","csi-hostpath-provisioning-7066":"k8s-agentpool-24944505-vmss000000","csi-hostpath-provisioning-716":"k8s-agentpool-24944505-vmss000000","csi-hostpath-provisioning-833":"k8s-agentpool-24944505-vmss000000","csi-hostpath-volume-9286":"k8s-agentpool-24944505-vmss000000","csi-hostpath-volume-expand-4114":"k8s-agentpool-24944505-vmss000000","csi-hostpath-volume-expand-4561":"k8s-agentpool-24944505-vmss000000","csi-hostpath-volume-expand-9975":"k8s-agentpool-24944505-vmss000000","csi-mock-csi-mock-volumes-2863":"csi-mock-csi-mock-volumes-2863","csi-mock-csi-mock-volumes-5360":"csi-mock-csi-mock-volumes-5360","csi-mock-csi-mock-volumes-6481":"csi-mock-csi-mock-volumes-6481","csi-mock-csi-mock-volumes-6951":"csi-mock-csi-mock-volumes-6951","csi-mock-csi-mock-volumes-8439":"csi-mock-csi-mock-volumes-8439","csi-mock-csi-mock-volumes-8606":"csi-mock-csi-mock-volumes-8606"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-24944505-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16797569024 0} {<nil>} 16403876Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16011137024 0} {<nil>} 15635876Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 21:18:20 +0000 UTC,LastTransitionTime:2019-11-13 20:42:03 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 21:18:20 +0000 UTC,LastTransitionTime:2019-11-13 20:42:03 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 21:18:20 +0000 UTC,LastTransitionTime:2019-11-13 20:42:03 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 21:18:20 +0000 UTC,LastTransitionTime:2019-11-13 20:42:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.248.0.4,},NodeAddress{Type:Hostname,Address:k8s-agentpool-24944505-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:0a2e8938ebfc440180ed5c0b5a9edbe7,SystemUUID:8075A1F3-D764-7248-9570-A764E51D7FF2,BootID:4f2b3cbe-c61c-4c6c-93ab-18fca0df5bc5,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[quay.io/kubernetes_incubator/nfs-provisioner@sha256:df762117e3c891f2d2ddff46ecb0776ba1f9f3c44cfd7739b0683bcd7a7954a8 quay.io/kubernetes_incubator/nfs-provisioner:v2.2.2],SizeBytes:391772778,},ContainerImage{Names:[gcr.io/google-samples/gb-frontend@sha256:35cb427341429fac3df10ff74600ea73e8ec0754d78f9ce89e0b4f3d70d53ba6 gcr.io/google-samples/gb-frontend:v6],SizeBytes:373099368,},ContainerImage{Names:[k8s.gcr.io/etcd@sha256:12c2c5e5731c3bcd56e6f1c05c0f9198b6f06793fa7fca2fb43aab9622dc4afa k8s.gcr.io/etcd:3.3.15],SizeBytes:246640776,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/volume/nfs@sha256:c2ad734346f608a5f7d69cfded93c4e8094069320657bd372d12ba21dea3ea71 gcr.io/kubernetes-e2e-test-images/volume/nfs:1.0],SizeBytes:225358913,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/jessie-dnsutils@sha256:ad583e33cb284f7ef046673809b146ec4053cda19b54a85d2b180a86169715eb gcr.io/kubernetes-e2e-test-images/jessie-dnsutils:1.0],SizeBytes:195659796,},ContainerImage{Names:[httpd@sha256:addd70e4ee83f3bc9a4c1c7c41e37927ba47faf639312fc936df3afad7926f5a httpd:2.4.39-alpine],SizeBytes:126894770,},ContainerImage{Names:[httpd@sha256:eb8ccf084cf3e80eece1add239effefd171eb39adbc154d33c14260d905d4060 httpd:2.4.38-alpine],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1],SizeBytes:121711221,},ContainerImage{Names:[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0],SizeBytes:71821984,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/sample-apiserver@sha256:1bafcc6fb1aa990b487850adba9cadc020e42d7905aa8a30481182a477ba24b0 gcr.io/kubernetes-e2e-test-images/sample-apiserver:1.10],SizeBytes:61365829,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727 gcr.io/kubernetes-e2e-test-images/agnhost:2.6],SizeBytes:57345321,},ContainerImage{Names:[quay.io/k8scsi/csi-provisioner@sha256:0efcb424f1dde9b9fb11a1a14f2e48ab47e1c3f08bc3a929990dcfcb1f7ab34f quay.io/k8scsi/csi-provisioner:v1.4.0-rc1],SizeBytes:54431016,},ContainerImage{Names:[quay.io/k8scsi/csi-snapshotter@sha256:e3d3e742e32d00488fdb401045b9b1d033d7ca0ab6e760f77b24750fc95e5f70 quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1],SizeBytes:51703561,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0],SizeBytes:46226754,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:0aba670b4d9d6b2e720bbf575d733156c676b693ca26501235444490300db838 quay.io/k8scsi/csi-attacher:v1.1.0],SizeBytes:42839085,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:7d46fb6eb8b890dc546029d1565d502b4a1d974d33625c6ee2bc7991b77fc1a1 quay.io/k8scsi/csi-resizer:v0.2.0],SizeBytes:42817100,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:f315c9042e56def3c05c6b04fe79ec9da6d39ddc557ca365a76cf35964ea08b6 quay.io/k8scsi/csi-resizer:v0.1.0],SizeBytes:42623056,},ContainerImage{Names:[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1],SizeBytes:42541759,},ContainerImage{Names:[gcr.io/google-containers/debian-base@sha256:6966a0aedd7592c18ff2dd803c08bd85780ee19f5e3a2e7cf908a4cd837afcde gcr.io/google-containers/debian-base:0.4.1],SizeBytes:42323657,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nonroot@sha256:d4ede5c74517090b6686219059118ed178cf4620f5db8781b32f806bb1e7395b gcr.io/kubernetes-e2e-test-images/nonroot:1.0],SizeBytes:42321438,},ContainerImage{Names:[redis@sha256:50899ea1ceed33fa03232f3ac57578a424faa1742c1ac9c7a7bdb95cdf19b858 redis:5.0.5-alpine],SizeBytes:29331594,},ContainerImage{Names:[quay.io/k8scsi/hostpathplugin@sha256:b4826e492fc1762fceaf9726f41575ca0a4567864d3d235da874818de18039de quay.io/k8scsi/hostpathplugin:v1.2.0-rc5],SizeBytes:28761497,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7],SizeBytes:22909487,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/echoserver@sha256:e9ba514b896cdf559eef8788b66c2c3ee55f3572df617647b4b0d8b6bf81cf19 gcr.io/kubernetes-e2e-test-images/echoserver:2.2],SizeBytes:21692741,},ContainerImage{Names:[quay.io/k8scsi/mock-driver@sha256:e0eed916b7d970bad2b7d9875f9ad16932f987f0f3d91ec5d86da68b0b5cc9d1 quay.io/k8scsi/mock-driver:v2.1.0],SizeBytes:16226335,},ContainerImage{Names:[nginx@sha256:485b610fefec7ff6c463ced9623314a04ed67e3945b9c08d7e53a47f6d108dc7 nginx:1.14-alpine],SizeBytes:16032814,},ContainerImage{Names:[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0],SizeBytes:15815995,},ContainerImage{Names:[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0],SizeBytes:14967303,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/dnsutils@sha256:2abeee84efb79c14d731966e034af33bf324d3b26ca28497555511ff094b3ddd gcr.io/kubernetes-e2e-test-images/dnsutils:1.1],SizeBytes:9349974,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nonewprivs@sha256:10066e9039219449fe3c81f38fe01928f87914150768ab81b62a468e51fa7411 gcr.io/kubernetes-e2e-test-images/nonewprivs:1.0],SizeBytes:6757579,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nautilus@sha256:33a732d4c42a266912a5091598a0f07653c9134db4b8d571690d8afd509e0bfc gcr.io/kubernetes-e2e-test-images/nautilus:1.0],SizeBytes:4753501,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/kitten@sha256:bcbc4875c982ab39aa7c4f6acf4a287f604e996d9f34a3fbda8c3d1a7457d1f6 gcr.io/kubernetes-e2e-test-images/kitten:1.0],SizeBytes:4747037,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/test-webserver@sha256:7f93d6e32798ff28bc6289254d0c2867fe2c849c8e46edc50f8624734309812e gcr.io/kubernetes-e2e-test-images/test-webserver:1.0],SizeBytes:4732240,},ContainerImage{Names:[alpine@sha256:8421d9a84432575381bfabd248f1eb56f3aa21d9d7cd2511583c68c9b7511d10 alpine:3.7],SizeBytes:4206494,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 gcr.io/kubernetes-e2e-test-images/mounttest:1.0],SizeBytes:1563521,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest-user@sha256:17319ca525ee003681fccf7e8c6b1b910ff4f49b653d939ac7f9b6e7c463933d gcr.io/kubernetes-e2e-test-images/mounttest-user:1.0],SizeBytes:1450451,},ContainerImage{Names:[busybox@sha256:1303dbf110c57f3edf68d9f5a16c082ec06c4cf7604831669faf2c712260b5a0 busybox:latest],SizeBytes:1219790,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea k8s.gcr.io/pause-amd64:3.1 k8s.gcr.io/pause:3.1],SizeBytes:742472,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-provisioning-833^f695f17e-065a-11ea-913d-000d3ac2cd71],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-provisioning-833^f695f17e-065a-11ea-913d-000d3ac2cd71,DevicePath:,},},Config:nil,},} Nov 13 21:18:31.713: INFO: Logging kubelet events for node k8s-agentpool-24944505-vmss000000 Nov 13 21:18:31.767: INFO: Logging pods the kubelet thinks is on node k8s-agentpool-24944505-vmss000000 Nov 13 21:18:31.824: INFO: csi-hostpath-attacher-0 started at 2019-11-13 21:14:59 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:31.824: INFO: Container csi-attacher ready: true, restart count 0 Nov 13 21:18:31.824: INFO: external-provisioner-cm8f2 started at 2019-11-13 21:17:10 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:31.824: INFO: Container nfs-provisioner ready: true, restart count 0 Nov 13 21:18:31.824: INFO: recycler-for-nfs-kbrgv started at 2019-11-13 21:18:27 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:31.824: INFO: Container pv-recycler ready: false, restart count 0 Nov 13 21:18:31.824: INFO: affinity-nodeport-xjb88 started at 2019-11-13 21:16:59 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:31.824: INFO: Container affinity-nodeport ready: true, restart count 0 Nov 13 21:18:31.824: INFO: keyvault-flexvolume-sn47m started at 2019-11-13 20:42:59 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:31.824: INFO: Container keyvault-flexvolume ready: true, restart count 0 Nov 13 21:18:31.824: INFO: blobfuse-flexvol-installer-drhkj started at 2019-11-13 20:42:59 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:31.824: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 Nov 13 21:18:31.824: INFO: csi-hostpath-resizer-0 started at 2019-11-13 21:15:04 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:31.824: INFO: Container csi-resizer ready: true, restart count 0 Nov 13 21:18:31.824: INFO: kube-proxy-h9n4c started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:31.824: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 21:18:31.824: INFO: kubernetes-dashboard-65966766b9-lq4lm started at 2019-11-13 20:42:59 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:31.824: INFO: Container kubernetes-dashboard ready: true, restart count 0 Nov 13 21:18:31.824: INFO: affinity-nodeport-hxwfs started at 2019-11-13 21:16:58 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:31.824: INFO: Container affinity-nodeport ready: true, restart count 0 Nov 13 21:18:31.824: INFO: pfpod started at 2019-11-13 21:17:14 +0000 UTC (0+2 container statuses recorded) Nov 13 21:18:31.824: INFO: Container portforwardtester ready: false, restart count 0 Nov 13 21:18:31.824: INFO: Container readiness ready: false, restart count 0 Nov 13 21:18:31.824: INFO: pod-handle-http-request started at 2019-11-13 21:15:55 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:31.824: INFO: Container pod-handle-http-request ready: true, restart count 0 Nov 13 21:18:31.824: INFO: execpod-affinitytrc2b started at 2019-11-13 21:17:56 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:31.824: INFO: Container agnhost-pause ready: true, restart count 0 Nov 13 21:18:31.824: INFO: pod-subpath-test-local-preprovisionedpv-zbxt started at 2019-11-13 21:17:36 +0000 UTC (1+1 container statuses recorded) Nov 13 21:18:31.824: INFO: Init container init-volume-local-preprovisionedpv-zbxt ready: true, restart count 0 Nov 13 21:18:31.824: INFO: Container test-container-subpath-local-preprovisionedpv-zbxt ready: true, restart count 0 Nov 13 21:18:31.824: INFO: pfpod started at 2019-11-13 21:17:38 +0000 UTC (0+2 container statuses recorded) Nov 13 21:18:31.824: INFO: Container portforwardtester ready: true, restart count 0 Nov 13 21:18:31.824: INFO: Container readiness ready: false, restart count 0 Nov 13 21:18:31.824: INFO: pod-submit-remove-b5ae053d-573d-4159-a262-23a1fd285799 started at 2019-11-13 21:17:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:31.824: INFO: Container nginx ready: false, restart count 0 Nov 13 21:18:31.824: INFO: metrics-server-58ff8c5ddf-r2d7f started at 2019-11-13 20:42:59 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:31.824: INFO: Container metrics-server ready: true, restart count 0 Nov 13 21:18:31.824: INFO: hostexec-k8s-agentpool-24944505-vmss000000 started at 2019-11-13 21:16:49 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:31.824: INFO: Container agnhost ready: true, restart count 0 Nov 13 21:18:31.824: INFO: azure-ip-masq-agent-qksp6 started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:31.824: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 21:18:31.824: INFO: tiller-deploy-7559b6b885-qxwkm started at 2019-11-13 20:42:59 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:31.824: INFO: Container tiller ready: true, restart count 0 Nov 13 21:18:31.824: INFO: externalname-service-4qp7f started at 2019-11-13 21:17:20 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:31.824: INFO: Container externalname-service ready: true, restart count 0 Nov 13 21:18:31.824: INFO: csi-hostpathplugin-0 started at 2019-11-13 21:16:34 +0000 UTC (0+3 container statuses recorded) Nov 13 21:18:31.824: INFO: Container hostpath ready: true, restart count 0 Nov 13 21:18:31.824: INFO: Container liveness-probe ready: true, restart count 0 Nov 13 21:18:31.824: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 13 21:18:31.824: INFO: pod-subpath-test-csi-hostpath-dynamicpv-stmr started at 2019-11-13 21:17:19 +0000 UTC (1+1 container statuses recorded) Nov 13 21:18:31.824: INFO: Init container init-volume-csi-hostpath-dynamicpv-stmr ready: true, restart count 0 Nov 13 21:18:31.824: INFO: Container test-container-subpath-csi-hostpath-dynamicpv-stmr ready: true, restart count 0 Nov 13 21:18:31.824: INFO: nfs-server started at 2019-11-13 21:17:01 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:31.824: INFO: Container nfs-server ready: true, restart count 0 Nov 13 21:18:31.824: INFO: pod-subpath-test-nfs-dynamicpv-zqjt started at 2019-11-13 21:17:53 +0000 UTC (2+1 container statuses recorded) Nov 13 21:18:31.824: INFO: Init container init-volume-nfs-dynamicpv-zqjt ready: true, restart count 0 Nov 13 21:18:31.824: INFO: Init container test-init-subpath-nfs-dynamicpv-zqjt ready: false, restart count 0 Nov 13 21:18:31.824: INFO: Container test-container-subpath-nfs-dynamicpv-zqjt ready: false, restart count 0 Nov 13 21:18:31.824: INFO: csi-hostpath-provisioner-0 started at 2019-11-13 21:15:02 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:31.824: INFO: Container csi-provisioner ready: true, restart count 0 Nov 13 21:18:31.824: INFO: csi-snapshotter-0 started at 2019-11-13 21:15:05 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:31.824: INFO: Container csi-snapshotter ready: true, restart count 0 W1113 21:18:31.885550 92630 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 21:18:33.214: INFO: Latency metrics for node k8s-agentpool-24944505-vmss000000 Nov 13 21:18:33.214: INFO: Logging node info for node k8s-agentpool-24944505-vmss000001 Nov 13 21:18:33.265: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool-24944505-vmss000001 /api/v1/nodes/k8s-agentpool-24944505-vmss000001 da675d2e-0aad-4a8f-b94d-1311fb000563 53620 0 2019-11-13 20:42:24 +0000 UTC <nil> <nil> map[agentpool:agentpool beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool-24944505-vmss000001 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: storageprofile:managed storagetier:Premium_LRS] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-351":"k8s-agentpool-24944505-vmss000001","csi-hostpath-ephemeral-4187":"k8s-agentpool-24944505-vmss000001","csi-hostpath-ephemeral-5584":"k8s-agentpool-24944505-vmss000001","csi-hostpath-provisioning-1638":"k8s-agentpool-24944505-vmss000001","csi-hostpath-provisioning-225":"k8s-agentpool-24944505-vmss000001","csi-hostpath-provisioning-7099":"k8s-agentpool-24944505-vmss000001","csi-hostpath-volume-6291":"k8s-agentpool-24944505-vmss000001","csi-hostpath-volume-expand-7524":"k8s-agentpool-24944505-vmss000001","csi-hostpath-volume-expand-8458":"k8s-agentpool-24944505-vmss000001","csi-mock-csi-mock-volumes-2938":"csi-mock-csi-mock-volumes-2938","csi-mock-csi-mock-volumes-4469":"csi-mock-csi-mock-volumes-4469","csi-mock-csi-mock-volumes-4959":"csi-mock-csi-mock-volumes-4959","csi-mock-csi-mock-volumes-597":"csi-mock-csi-mock-volumes-597","csi-mock-csi-mock-volumes-8812":"csi-mock-csi-mock-volumes-8812","csi-mock-csi-mock-volumes-9088":"csi-mock-csi-mock-volumes-9088","csi-mock-csi-mock-volumes-9179":"csi-mock-csi-mock-volumes-9179"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-24944505-vmss/virtualMachines/1,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16797569024 0} {<nil>} 16403876Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16011137024 0} {<nil>} 15635876Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 21:18:03 +0000 UTC,LastTransitionTime:2019-11-13 20:42:24 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 21:18:03 +0000 UTC,LastTransitionTime:2019-11-13 20:42:24 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 21:18:03 +0000 UTC,LastTransitionTime:2019-11-13 20:42:24 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 21:18:03 +0000 UTC,LastTransitionTime:2019-11-13 20:42:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.248.0.5,},NodeAddress{Type:Hostname,Address:k8s-agentpool-24944505-vmss000001,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:6ed41b2c4b884fccaf94748594531e06,SystemUUID:9CE5C2E3-7C34-C64A-9734-2477E19937B7,BootID:306ac3f6-a23b-423e-83c9-e3d5f3938f6c,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[quay.io/kubernetes_incubator/nfs-provisioner@sha256:df762117e3c891f2d2ddff46ecb0776ba1f9f3c44cfd7739b0683bcd7a7954a8 quay.io/kubernetes_incubator/nfs-provisioner:v2.2.2],SizeBytes:391772778,},ContainerImage{Names:[gluster/glusterdynamic-provisioner@sha256:90067cb05a7d217651e84576935934fd9dcff8e6bcdcbaa416bbf36fcd09dbd1 gluster/glusterdynamic-provisioner:v1.0],SizeBytes:373281573,},ContainerImage{Names:[gcr.io/google-samples/gb-frontend@sha256:35cb427341429fac3df10ff74600ea73e8ec0754d78f9ce89e0b4f3d70d53ba6 gcr.io/google-samples/gb-frontend:v6],SizeBytes:373099368,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/volume/nfs@sha256:c2ad734346f608a5f7d69cfded93c4e8094069320657bd372d12ba21dea3ea71 gcr.io/kubernetes-e2e-test-images/volume/nfs:1.0],SizeBytes:225358913,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/jessie-dnsutils@sha256:ad583e33cb284f7ef046673809b146ec4053cda19b54a85d2b180a86169715eb gcr.io/kubernetes-e2e-test-images/jessie-dnsutils:1.0],SizeBytes:195659796,},ContainerImage{Names:[httpd@sha256:addd70e4ee83f3bc9a4c1c7c41e37927ba47faf639312fc936df3afad7926f5a httpd:2.4.39-alpine],SizeBytes:126894770,},ContainerImage{Names:[httpd@sha256:eb8ccf084cf3e80eece1add239effefd171eb39adbc154d33c14260d905d4060 httpd:2.4.38-alpine],SizeBytes:123781643,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727 gcr.io/kubernetes-e2e-test-images/agnhost:2.6],SizeBytes:57345321,},ContainerImage{Names:[quay.io/k8scsi/csi-provisioner@sha256:0efcb424f1dde9b9fb11a1a14f2e48ab47e1c3f08bc3a929990dcfcb1f7ab34f quay.io/k8scsi/csi-provisioner:v1.4.0-rc1],SizeBytes:54431016,},ContainerImage{Names:[quay.io/k8scsi/csi-snapshotter@sha256:e3d3e742e32d00488fdb401045b9b1d033d7ca0ab6e760f77b24750fc95e5f70 quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1],SizeBytes:51703561,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0],SizeBytes:46226754,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:0aba670b4d9d6b2e720bbf575d733156c676b693ca26501235444490300db838 quay.io/k8scsi/csi-attacher:v1.1.0],SizeBytes:42839085,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:7d46fb6eb8b890dc546029d1565d502b4a1d974d33625c6ee2bc7991b77fc1a1 quay.io/k8scsi/csi-resizer:v0.2.0],SizeBytes:42817100,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:f315c9042e56def3c05c6b04fe79ec9da6d39ddc557ca365a76cf35964ea08b6 quay.io/k8scsi/csi-resizer:v0.1.0],SizeBytes:42623056,},ContainerImage{Names:[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0],SizeBytes:42488424,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nonroot@sha256:d4ede5c74517090b6686219059118ed178cf4620f5db8781b32f806bb1e7395b gcr.io/kubernetes-e2e-test-images/nonroot:1.0],SizeBytes:42321438,},ContainerImage{Names:[redis@sha256:50899ea1ceed33fa03232f3ac57578a424faa1742c1ac9c7a7bdb95cdf19b858 redis:5.0.5-alpine],SizeBytes:29331594,},ContainerImage{Names:[quay.io/k8scsi/hostpathplugin@sha256:b4826e492fc1762fceaf9726f41575ca0a4567864d3d235da874818de18039de quay.io/k8scsi/hostpathplugin:v1.2.0-rc5],SizeBytes:28761497,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7],SizeBytes:22909487,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/echoserver@sha256:e9ba514b896cdf559eef8788b66c2c3ee55f3572df617647b4b0d8b6bf81cf19 gcr.io/kubernetes-e2e-test-images/echoserver:2.2],SizeBytes:21692741,},ContainerImage{Names:[quay.io/k8scsi/mock-driver@sha256:e0eed916b7d970bad2b7d9875f9ad16932f987f0f3d91ec5d86da68b0b5cc9d1 quay.io/k8scsi/mock-driver:v2.1.0],SizeBytes:16226335,},ContainerImage{Names:[nginx@sha256:485b610fefec7ff6c463ced9623314a04ed67e3945b9c08d7e53a47f6d108dc7 nginx:1.14-alpine],SizeBytes:16032814,},ContainerImage{Names:[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0],SizeBytes:15815995,},ContainerImage{Names:[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0],SizeBytes:14967303,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/dnsutils@sha256:2abeee84efb79c14d731966e034af33bf324d3b26ca28497555511ff094b3ddd gcr.io/kubernetes-e2e-test-images/dnsutils:1.1],SizeBytes:9349974,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nautilus@sha256:33a732d4c42a266912a5091598a0f07653c9134db4b8d571690d8afd509e0bfc gcr.io/kubernetes-e2e-test-images/nautilus:1.0],SizeBytes:4753501,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/kitten@sha256:bcbc4875c982ab39aa7c4f6acf4a287f604e996d9f34a3fbda8c3d1a7457d1f6 gcr.io/kubernetes-e2e-test-images/kitten:1.0],SizeBytes:4747037,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/test-webserver@sha256:7f93d6e32798ff28bc6289254d0c2867fe2c849c8e46edc50f8624734309812e gcr.io/kubernetes-e2e-test-images/test-webserver:1.0],SizeBytes:4732240,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:4206620,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 gcr.io/kubernetes-e2e-test-images/mounttest:1.0],SizeBytes:1563521,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest-user@sha256:17319ca525ee003681fccf7e8c6b1b910ff4f49b653d939ac7f9b6e7c463933d gcr.io/kubernetes-e2e-test-images/mounttest-user:1.0],SizeBytes:1450451,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea k8s.gcr.io/pause-amd64:3.1 k8s.gcr.io/pause:3.1],SizeBytes:742472,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-9179^4],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-9179^4,DevicePath:,},},Config:nil,},} Nov 13 21:18:33.266: INFO: Logging kubelet events for node k8s-agentpool-24944505-vmss000001 Nov 13 21:18:33.321: INFO: Logging pods the kubelet thinks is on node k8s-agentpool-24944505-vmss000001 Nov 13 21:18:33.492: INFO: affinity-nodeport-6dzh9 started at 2019-11-13 21:16:59 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:33.492: INFO: Container affinity-nodeport ready: true, restart count 0 Nov 13 21:18:33.492: INFO: local-client started at 2019-11-13 21:18:07 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:33.492: INFO: Container local-client ready: false, restart count 0 Nov 13 21:18:33.492: INFO: local-client started at 2019-11-13 21:17:32 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:33.492: INFO: Container local-client ready: true, restart count 0 Nov 13 21:18:33.492: INFO: coredns-87f5d796-jd8bc started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:33.492: INFO: Container coredns ready: true, restart count 0 Nov 13 21:18:33.492: INFO: csi-snapshotter-0 started at 2019-11-13 21:15:23 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:33.492: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 13 21:18:33.492: INFO: inline-volume-tester-qvfqv started at 2019-11-13 21:15:24 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:33.492: INFO: Container csi-volume-tester ready: false, restart count 0 Nov 13 21:18:33.492: INFO: keyvault-flexvolume-7phpr started at 2019-11-13 20:42:58 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:33.492: INFO: Container keyvault-flexvolume ready: true, restart count 0 Nov 13 21:18:33.492: INFO: blobfuse-flexvol-installer-cpm6h started at 2019-11-13 20:42:58 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:33.492: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 Nov 13 21:18:33.492: INFO: csi-mockplugin-0 started at 2019-11-13 21:15:00 +0000 UTC (0+3 container statuses recorded) Nov 13 21:18:33.492: INFO: Container csi-provisioner ready: true, restart count 0 Nov 13 21:18:33.492: INFO: Container driver-registrar ready: true, restart count 0 Nov 13 21:18:33.492: INFO: Container mock ready: true, restart count 0 Nov 13 21:18:33.492: INFO: csi-mockplugin-attacher-0 started at 2019-11-13 21:17:07 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:33.492: INFO: Container csi-attacher ready: true, restart count 0 Nov 13 21:18:33.492: INFO: azure-ip-masq-agent-wwpg7 started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:33.492: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 21:18:33.492: INFO: hostexec-k8s-agentpool-24944505-vmss000001 started at 2019-11-13 21:15:01 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:33.492: INFO: Container agnhost ready: true, restart count 0 Nov 13 21:18:33.492: INFO: kube-proxy-7bpf7 started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:33.492: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 21:18:33.492: INFO: csi-hostpath-attacher-0 started at 2019-11-13 21:15:16 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:33.492: INFO: Container csi-attacher ready: true, restart count 0 Nov 13 21:18:33.492: INFO: csi-hostpathplugin-0 started at 2019-11-13 21:15:21 +0000 UTC (0+3 container statuses recorded) Nov 13 21:18:33.492: INFO: Container hostpath ready: true, restart count 0 Nov 13 21:18:33.492: INFO: Container liveness-probe ready: true, restart count 0 Nov 13 21:18:33.492: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 13 21:18:33.492: INFO: externalname-service-kxgxn started at 2019-11-13 21:17:17 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:33.492: INFO: Container externalname-service ready: true, restart count 0 Nov 13 21:18:33.492: INFO: csi-hostpath-provisioner-0 started at 2019-11-13 21:15:22 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:33.492: INFO: Container csi-provisioner ready: true, restart count 0 Nov 13 21:18:33.492: INFO: pvc-volume-tester-jpmhl started at 2019-11-13 21:17:59 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:33.492: INFO: Container volume-tester ready: false, restart count 0 Nov 13 21:18:33.492: INFO: execpodgdrsf started at 2019-11-13 21:18:00 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:33.492: INFO: Container agnhost-pause ready: true, restart count 0 Nov 13 21:18:33.492: INFO: csi-hostpath-resizer-0 started at 2019-11-13 21:15:22 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:33.492: INFO: Container csi-resizer ready: true, restart count 0 Nov 13 21:18:33.492: INFO: hostexec-k8s-agentpool-24944505-vmss000001 started at 2019-11-13 21:13:43 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:33.492: INFO: Container agnhost ready: true, restart count 0 Nov 13 21:18:33.492: INFO: hostpath-client started at 2019-11-13 21:18:04 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:33.492: INFO: Container hostpath-client ready: true, restart count 0 Nov 13 21:18:33.492: INFO: hostexec-k8s-agentpool-24944505-vmss000001 started at 2019-11-13 21:17:56 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:33.492: INFO: Container agnhost ready: false, restart count 0 Nov 13 21:18:33.492: INFO: pod-subpath-test-hostpathsymlink-vjl4 started at 2019-11-13 21:18:32 +0000 UTC (1+2 container statuses recorded) Nov 13 21:18:33.492: INFO: Init container test-init-subpath-hostpathsymlink-vjl4 ready: false, restart count 0 Nov 13 21:18:33.492: INFO: Container test-container-subpath-hostpathsymlink-vjl4 ready: false, restart count 0 Nov 13 21:18:33.492: INFO: Container test-container-volume-hostpathsymlink-vjl4 ready: false, restart count 0 Nov 13 21:18:33.492: INFO: csi-mockplugin-0 started at 2019-11-13 21:17:05 +0000 UTC (0+3 container statuses recorded) Nov 13 21:18:33.492: INFO: Container csi-provisioner ready: true, restart count 0 Nov 13 21:18:33.492: INFO: Container driver-registrar ready: true, restart count 0 Nov 13 21:18:33.492: INFO: Container mock ready: true, restart count 0 W1113 21:18:33.544837 92630 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 21:18:33.708: INFO: Latency metrics for node k8s-agentpool-24944505-vmss000001 Nov 13 21:18:33.708: INFO: Logging node info for node k8s-master-24944505-vmss000000 Nov 13 21:18:33.770: INFO: Node Info: &Node{ObjectMeta:{k8s-master-24944505-vmss000000 /api/v1/nodes/k8s-master-24944505-vmss000000 048fbae2-b97f-43a2-b471-398180136f12 53765 0 2019-11-13 20:42:13 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-24944505-vmss000000 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-24944505-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284887552 0} {<nil>} 7114148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6498455552 0} {<nil>} 6346148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 21:18:09 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 21:18:09 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 21:18:09 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 21:18:09 +0000 UTC,LastTransitionTime:2019-11-13 20:42:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.4,},NodeAddress{Type:Hostname,Address:k8s-master-24944505-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:7bd754c0dc8647518c3043bcae3a76cc,SystemUUID:FA5CFE99-8148-6C46-B4E6-5EA2A5DE9D2A,BootID:74f7b0a2-e85c-438e-a642-c5b32900bb28,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:3237a53b135d8692ade82eca845fd0259ecbfcbe05fad35400e7d5c2018c4abd k8sprow.azurecr.io/azure-cloud-controller-manager:1194710255657816064-e688ecb0],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 21:18:33.770: INFO: Logging kubelet events for node k8s-master-24944505-vmss000000 Nov 13 21:18:33.826: INFO: Logging pods the kubelet thinks is on node k8s-master-24944505-vmss000000 Nov 13 21:18:33.899: INFO: kube-scheduler-k8s-master-24944505-vmss000000 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:33.899: INFO: Container kube-scheduler ready: true, restart count 0 Nov 13 21:18:33.899: INFO: azure-ip-masq-agent-6pnst started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:33.899: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 21:18:33.899: INFO: kube-proxy-d9jjh started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:33.899: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 21:18:33.899: INFO: cloud-controller-manager-k8s-master-24944505-vmss000000 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:33.899: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 13 21:18:33.899: INFO: kube-addon-manager-k8s-master-24944505-vmss000000 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:33.899: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 13 21:18:33.899: INFO: kube-apiserver-k8s-master-24944505-vmss000000 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:33.899: INFO: Container kube-apiserver ready: true, restart count 0 Nov 13 21:18:33.899: INFO: kube-controller-manager-k8s-master-24944505-vmss000000 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:33.899: INFO: Container kube-controller-manager ready: true, restart count 0 W1113 21:18:33.951544 92630 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 21:18:34.073: INFO: Latency metrics for node k8s-master-24944505-vmss000000 Nov 13 21:18:34.073: INFO: Logging node info for node k8s-master-24944505-vmss000001 Nov 13 21:18:34.124: INFO: Node Info: &Node{ObjectMeta:{k8s-master-24944505-vmss000001 /api/v1/nodes/k8s-master-24944505-vmss000001 a2619878-b119-44f9-b89f-4073bebbb56c 53671 0 2019-11-13 20:42:40 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-24944505-vmss000001 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.6.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-24944505-vmss/virtualMachines/1,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.6.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284883456 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6498451456 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:43:08 +0000 UTC,LastTransitionTime:2019-11-13 20:43:08 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 21:18:05 +0000 UTC,LastTransitionTime:2019-11-13 20:42:34 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 21:18:05 +0000 UTC,LastTransitionTime:2019-11-13 20:42:34 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 21:18:05 +0000 UTC,LastTransitionTime:2019-11-13 20:42:34 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 21:18:05 +0000 UTC,LastTransitionTime:2019-11-13 20:42:40 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.5,},NodeAddress{Type:Hostname,Address:k8s-master-24944505-vmss000001,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:c866a7c1ed64481990416e24387dd317,SystemUUID:EBE64557-2AC8-5740-9FFE-AB9F12412FBC,BootID:3dbb0e0e-0479-4821-bc3f-a8269d250e4c,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:3237a53b135d8692ade82eca845fd0259ecbfcbe05fad35400e7d5c2018c4abd k8sprow.azurecr.io/azure-cloud-controller-manager:1194710255657816064-e688ecb0],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 21:18:34.162: INFO: Logging kubelet events for node k8s-master-24944505-vmss000001 Nov 13 21:18:34.216: INFO: Logging pods the kubelet thinks is on node k8s-master-24944505-vmss000001 Nov 13 21:18:34.289: INFO: azure-ip-masq-agent-bs52d started at 2019-11-13 20:42:41 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:34.289: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 21:18:34.289: INFO: kube-scheduler-k8s-master-24944505-vmss000001 started at 2019-11-13 20:42:28 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:34.289: INFO: Container kube-scheduler ready: true, restart count 0 Nov 13 21:18:34.289: INFO: cloud-controller-manager-k8s-master-24944505-vmss000001 started at 2019-11-13 20:42:28 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:34.289: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 13 21:18:34.289: INFO: kube-addon-manager-k8s-master-24944505-vmss000001 started at 2019-11-13 20:42:28 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:34.289: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 13 21:18:34.289: INFO: kube-apiserver-k8s-master-24944505-vmss000001 started at 2019-11-13 20:42:28 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:34.289: INFO: Container kube-apiserver ready: true, restart count 0 Nov 13 21:18:34.289: INFO: kube-controller-manager-k8s-master-24944505-vmss000001 started at 2019-11-13 20:42:28 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:34.289: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 13 21:18:34.289: INFO: kube-proxy-w4bz5 started at 2019-11-13 20:42:41 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:34.290: INFO: Container kube-proxy ready: true, restart count 0 W1113 21:18:34.342008 92630 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 21:18:34.459: INFO: Latency metrics for node k8s-master-24944505-vmss000001 Nov 13 21:18:34.459: INFO: Logging node info for node k8s-master-24944505-vmss000002 Nov 13 21:18:34.511: INFO: Node Info: &Node{ObjectMeta:{k8s-master-24944505-vmss000002 /api/v1/nodes/k8s-master-24944505-vmss000002 f3b05267-a009-4dba-9f97-4f772e48707e 53761 0 2019-11-13 20:42:13 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-24944505-vmss000002 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.4.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-24944505-vmss/virtualMachines/2,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284887552 0} {<nil>} 7114148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6498455552 0} {<nil>} 6346148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 21:18:08 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 21:18:08 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 21:18:08 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 21:18:08 +0000 UTC,LastTransitionTime:2019-11-13 20:42:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.6,},NodeAddress{Type:Hostname,Address:k8s-master-24944505-vmss000002,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:fc1777f37ced4b9c97acb9b30e276c4f,SystemUUID:E9DF19DC-D02C-714C-A7D6-0BA6CE959C55,BootID:622958a8-d45d-4edf-9266-4dbb759e2359,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:3237a53b135d8692ade82eca845fd0259ecbfcbe05fad35400e7d5c2018c4abd k8sprow.azurecr.io/azure-cloud-controller-manager:1194710255657816064-e688ecb0],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 21:18:34.512: INFO: Logging kubelet events for node k8s-master-24944505-vmss000002 Nov 13 21:18:34.566: INFO: Logging pods the kubelet thinks is on node k8s-master-24944505-vmss000002 Nov 13 21:18:34.643: INFO: kube-addon-manager-k8s-master-24944505-vmss000002 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:34.643: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 13 21:18:34.643: INFO: kube-apiserver-k8s-master-24944505-vmss000002 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:34.643: INFO: Container kube-apiserver ready: true, restart count 0 Nov 13 21:18:34.643: INFO: kube-controller-manager-k8s-master-24944505-vmss000002 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:34.643: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 13 21:18:34.643: INFO: kube-scheduler-k8s-master-24944505-vmss000002 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:34.643: INFO: Container kube-scheduler ready: true, restart count 0 Nov 13 21:18:34.643: INFO: cloud-controller-manager-k8s-master-24944505-vmss000002 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:34.643: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 13 21:18:34.643: INFO: azure-ip-masq-agent-6skf2 started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:34.643: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 21:18:34.643: INFO: kube-proxy-xwz78 started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:34.643: INFO: Container kube-proxy ready: true, restart count 0 W1113 21:18:34.696574 92630 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 21:18:34.818: INFO: Latency metrics for node k8s-master-24944505-vmss000002 Nov 13 21:18:34.818: INFO: Logging node info for node k8s-master-24944505-vmss000003 Nov 13 21:18:34.871: INFO: Node Info: &Node{ObjectMeta:{k8s-master-24944505-vmss000003 /api/v1/nodes/k8s-master-24944505-vmss000003 888cc19e-c62d-4945-a425-97d14e38a0b8 53542 0 2019-11-13 20:42:35 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-24944505-vmss000003 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.5.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-24944505-vmss/virtualMachines/3,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.5.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284887552 0} {<nil>} 7114148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6498455552 0} {<nil>} 6346148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 21:18:01 +0000 UTC,LastTransitionTime:2019-11-13 20:42:29 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 21:18:01 +0000 UTC,LastTransitionTime:2019-11-13 20:42:29 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 21:18:01 +0000 UTC,LastTransitionTime:2019-11-13 20:42:29 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 21:18:01 +0000 UTC,LastTransitionTime:2019-11-13 20:42:35 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.7,},NodeAddress{Type:Hostname,Address:k8s-master-24944505-vmss000003,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:2da5fe5a442b4e178a2b62ba91366e4b,SystemUUID:D2D1E816-D70E-3D40-B959-B00A2B76A83A,BootID:c5929aa1-3dac-45d5-bb2c-174c73c7300e,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:3237a53b135d8692ade82eca845fd0259ecbfcbe05fad35400e7d5c2018c4abd k8sprow.azurecr.io/azure-cloud-controller-manager:1194710255657816064-e688ecb0],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 21:18:34.872: INFO: Logging kubelet events for node k8s-master-24944505-vmss000003 Nov 13 21:18:34.926: INFO: Logging pods the kubelet thinks is on node k8s-master-24944505-vmss000003 Nov 13 21:18:34.997: INFO: kube-apiserver-k8s-master-24944505-vmss000003 started at 2019-11-13 20:42:23 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:34.997: INFO: Container kube-apiserver ready: true, restart count 0 Nov 13 21:18:34.997: INFO: kube-controller-manager-k8s-master-24944505-vmss000003 started at 2019-11-13 20:42:23 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:34.998: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 13 21:18:34.998: INFO: kube-scheduler-k8s-master-24944505-vmss000003 started at 2019-11-13 20:42:23 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:34.998: INFO: Container kube-scheduler ready: true, restart count 0 Nov 13 21:18:34.998: INFO: cloud-controller-manager-k8s-master-24944505-vmss000003 started at 2019-11-13 20:42:44 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:34.998: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 13 21:18:34.998: INFO: azure-ip-masq-agent-nlqss started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:34.998: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 21:18:34.998: INFO: kube-proxy-kgh5h started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:34.998: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 21:18:34.998: INFO: kube-addon-manager-k8s-master-24944505-vmss000003 started at 2019-11-13 20:42:22 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:34.998: INFO: Container kube-addon-manager ready: true, restart count 0 W1113 21:18:35.049910 92630 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 21:18:35.167: INFO: Latency metrics for node k8s-master-24944505-vmss000003 Nov 13 21:18:35.167: INFO: Logging node info for node k8s-master-24944505-vmss000004 Nov 13 21:18:35.231: INFO: Node Info: &Node{ObjectMeta:{k8s-master-24944505-vmss000004 /api/v1/nodes/k8s-master-24944505-vmss000004 a26d844f-0612-406b-9a65-7e2ac952a2d5 53764 0 2019-11-13 20:42:14 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-24944505-vmss000004 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-24944505-vmss/virtualMachines/4,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284887552 0} {<nil>} 7114148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6498455552 0} {<nil>} 6346148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 21:18:09 +0000 UTC,LastTransitionTime:2019-11-13 20:42:09 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 21:18:09 +0000 UTC,LastTransitionTime:2019-11-13 20:42:09 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 21:18:09 +0000 UTC,LastTransitionTime:2019-11-13 20:42:09 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 21:18:09 +0000 UTC,LastTransitionTime:2019-11-13 20:42:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.8,},NodeAddress{Type:Hostname,Address:k8s-master-24944505-vmss000004,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:2208959aca1045d9b48fce8d784c1e41,SystemUUID:E253DB3A-3A4D-5648-8169-B2E2976CFB4B,BootID:09bc6101-2da9-4c8e-8274-43db8c5fe1cf,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:3237a53b135d8692ade82eca845fd0259ecbfcbe05fad35400e7d5c2018c4abd k8sprow.azurecr.io/azure-cloud-controller-manager:1194710255657816064-e688ecb0],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 21:18:35.231: INFO: Logging kubelet events for node k8s-master-24944505-vmss000004 Nov 13 21:18:35.286: INFO: Logging pods the kubelet thinks is on node k8s-master-24944505-vmss000004 Nov 13 21:18:35.366: INFO: cloud-controller-manager-k8s-master-24944505-vmss000004 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:35.366: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 13 21:18:35.366: INFO: kube-addon-manager-k8s-master-24944505-vmss000004 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:35.366: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 13 21:18:35.366: INFO: kube-apiserver-k8s-master-24944505-vmss000004 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:35.366: INFO: Container kube-apiserver ready: true, restart count 0 Nov 13 21:18:35.366: INFO: kube-controller-manager-k8s-master-24944505-vmss000004 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:35.366: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 13 21:18:35.366: INFO: kube-scheduler-k8s-master-24944505-vmss000004 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:35.366: INFO: Container kube-scheduler ready: true, restart count 0 Nov 13 21:18:35.366: INFO: azure-ip-masq-agent-xnw2h started at 2019-11-13 20:42:38 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:35.366: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 21:18:35.366: INFO: kube-proxy-fld6p started at 2019-11-13 20:42:38 +0000 UTC (0+1 container statuses recorded) Nov 13 21:18:35.366: INFO: Container kube-proxy ready: true, restart count 0 W1113 21:18:35.419283 92630 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 21:18:35.535: INFO: Latency metrics for node k8s-master-24944505-vmss000004 Nov 13 21:18:35.535: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "pods-6662" for this suite. Nov 13 21:18:41.753: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered Nov 13 21:18:43.481: INFO: namespace pods-6662 deletion completed in 7.892800232s
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[k8s\.io\]\s\[sig\-node\]\sPods\sExtended\s\[k8s\.io\]\sDelete\sGrace\sPeriod\sshould\sbe\ssubmitted\sand\sremoved\s\[Conformance\]$'
test/e2e/framework/framework.go:698 Nov 13 21:16:26.670: kubelet never observed the termination notice Unexpected error: <*errors.errorString | 0xc0000a10a0>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred test/e2e/node/pods.go:163from junit_28.xml
[BeforeEach] [k8s.io] [sig-node] Pods Extended test/e2e/framework/framework.go:151 �[1mSTEP�[0m: Creating a kubernetes client Nov 13 21:15:11.255: INFO: >>> kubeConfig: /workspace/aks880331199/kubeconfig/kubeconfig.westus2.json �[1mSTEP�[0m: Building a namespace api object, basename pods �[1mSTEP�[0m: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in pods-2091 �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace [BeforeEach] [k8s.io] Delete Grace Period test/e2e/node/pods.go:47 [It] should be submitted and removed [Conformance] test/e2e/framework/framework.go:698 �[1mSTEP�[0m: creating the pod �[1mSTEP�[0m: setting up selector �[1mSTEP�[0m: submitting the pod to kubernetes �[1mSTEP�[0m: verifying the pod is in kubernetes Nov 13 21:15:56.022: INFO: Asynchronously running '/usr/local/bin/kubectl kubectl --server=https://kubetest-74ff5bf7-0652-11ea-bade-36185e316941.westus2.cloudapp.azure.com --kubeconfig=/workspace/aks880331199/kubeconfig/kubeconfig.westus2.json proxy -p 0' �[1mSTEP�[0m: deleting the pod gracefully �[1mSTEP�[0m: verifying the kubelet observed the termination notice Nov 13 21:16:26.670: FAIL: kubelet never observed the termination notice Unexpected error: <*errors.errorString | 0xc0000a10a0>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred [AfterEach] [k8s.io] [sig-node] Pods Extended test/e2e/framework/framework.go:152 �[1mSTEP�[0m: Collecting events from namespace "pods-2091". �[1mSTEP�[0m: Found 6 events. Nov 13 21:16:26.723: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for pod-submit-remove-261e5ada-3b8a-4d79-ac63-dd17f887835a: {default-scheduler } Scheduled: Successfully assigned pods-2091/pod-submit-remove-261e5ada-3b8a-4d79-ac63-dd17f887835a to k8s-agentpool-24944505-vmss000001 Nov 13 21:16:26.723: INFO: At 2019-11-13 21:15:38 +0000 UTC - event for pod-submit-remove-261e5ada-3b8a-4d79-ac63-dd17f887835a: {kubelet k8s-agentpool-24944505-vmss000001} Pulling: Pulling image "docker.io/library/nginx:1.14-alpine" Nov 13 21:16:26.723: INFO: At 2019-11-13 21:15:40 +0000 UTC - event for pod-submit-remove-261e5ada-3b8a-4d79-ac63-dd17f887835a: {kubelet k8s-agentpool-24944505-vmss000001} Pulled: Successfully pulled image "docker.io/library/nginx:1.14-alpine" Nov 13 21:16:26.723: INFO: At 2019-11-13 21:15:42 +0000 UTC - event for pod-submit-remove-261e5ada-3b8a-4d79-ac63-dd17f887835a: {kubelet k8s-agentpool-24944505-vmss000001} Created: Created container nginx Nov 13 21:16:26.723: INFO: At 2019-11-13 21:15:50 +0000 UTC - event for pod-submit-remove-261e5ada-3b8a-4d79-ac63-dd17f887835a: {kubelet k8s-agentpool-24944505-vmss000001} Started: Started container nginx Nov 13 21:16:26.723: INFO: At 2019-11-13 21:16:08 +0000 UTC - event for pod-submit-remove-261e5ada-3b8a-4d79-ac63-dd17f887835a: {kubelet k8s-agentpool-24944505-vmss000001} Killing: Stopping container nginx Nov 13 21:16:26.774: INFO: POD NODE PHASE GRACE CONDITIONS Nov 13 21:16:26.774: INFO: pod-submit-remove-261e5ada-3b8a-4d79-ac63-dd17f887835a k8s-agentpool-24944505-vmss000001 Running 30s [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:15:11 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:15:55 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:15:55 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:15:11 +0000 UTC }] Nov 13 21:16:26.774: INFO: Nov 13 21:16:26.880: INFO: Logging node info for node k8s-agentpool-24944505-vmss000000 Nov 13 21:16:26.935: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool-24944505-vmss000000 /api/v1/nodes/k8s-agentpool-24944505-vmss000000 5bc7cf9e-80da-4e26-a2c2-20757e870d68 51664 0 2019-11-13 20:42:13 +0000 UTC <nil> <nil> map[agentpool:agentpool beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool-24944505-vmss000000 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: storageprofile:managed storagetier:Premium_LRS] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-1137":"k8s-agentpool-24944505-vmss000000","csi-hostpath-ephemeral-9135":"k8s-agentpool-24944505-vmss000000","csi-hostpath-provisioning-2585":"k8s-agentpool-24944505-vmss000000","csi-hostpath-provisioning-3303":"k8s-agentpool-24944505-vmss000000","csi-hostpath-provisioning-7066":"k8s-agentpool-24944505-vmss000000","csi-hostpath-provisioning-716":"k8s-agentpool-24944505-vmss000000","csi-hostpath-volume-9286":"k8s-agentpool-24944505-vmss000000","csi-hostpath-volume-expand-4114":"k8s-agentpool-24944505-vmss000000","csi-hostpath-volume-expand-4561":"k8s-agentpool-24944505-vmss000000","csi-hostpath-volume-expand-9975":"k8s-agentpool-24944505-vmss000000","csi-mock-csi-mock-volumes-2863":"csi-mock-csi-mock-volumes-2863","csi-mock-csi-mock-volumes-5360":"csi-mock-csi-mock-volumes-5360","csi-mock-csi-mock-volumes-6481":"csi-mock-csi-mock-volumes-6481","csi-mock-csi-mock-volumes-6951":"csi-mock-csi-mock-volumes-6951","csi-mock-csi-mock-volumes-8439":"csi-mock-csi-mock-volumes-8439","csi-mock-csi-mock-volumes-8606":"csi-mock-csi-mock-volumes-8606"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-24944505-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16797569024 0} {<nil>} 16403876Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16011137024 0} {<nil>} 15635876Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 21:16:10 +0000 UTC,LastTransitionTime:2019-11-13 20:42:03 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 21:16:10 +0000 UTC,LastTransitionTime:2019-11-13 20:42:03 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 21:16:10 +0000 UTC,LastTransitionTime:2019-11-13 20:42:03 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 21:16:10 +0000 UTC,LastTransitionTime:2019-11-13 20:42:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.248.0.4,},NodeAddress{Type:Hostname,Address:k8s-agentpool-24944505-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:0a2e8938ebfc440180ed5c0b5a9edbe7,SystemUUID:8075A1F3-D764-7248-9570-A764E51D7FF2,BootID:4f2b3cbe-c61c-4c6c-93ab-18fca0df5bc5,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[quay.io/kubernetes_incubator/nfs-provisioner@sha256:df762117e3c891f2d2ddff46ecb0776ba1f9f3c44cfd7739b0683bcd7a7954a8 quay.io/kubernetes_incubator/nfs-provisioner:v2.2.2],SizeBytes:391772778,},ContainerImage{Names:[gcr.io/google-samples/gb-frontend@sha256:35cb427341429fac3df10ff74600ea73e8ec0754d78f9ce89e0b4f3d70d53ba6 gcr.io/google-samples/gb-frontend:v6],SizeBytes:373099368,},ContainerImage{Names:[k8s.gcr.io/etcd@sha256:12c2c5e5731c3bcd56e6f1c05c0f9198b6f06793fa7fca2fb43aab9622dc4afa k8s.gcr.io/etcd:3.3.15],SizeBytes:246640776,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/volume/nfs@sha256:c2ad734346f608a5f7d69cfded93c4e8094069320657bd372d12ba21dea3ea71 gcr.io/kubernetes-e2e-test-images/volume/nfs:1.0],SizeBytes:225358913,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/jessie-dnsutils@sha256:ad583e33cb284f7ef046673809b146ec4053cda19b54a85d2b180a86169715eb gcr.io/kubernetes-e2e-test-images/jessie-dnsutils:1.0],SizeBytes:195659796,},ContainerImage{Names:[httpd@sha256:addd70e4ee83f3bc9a4c1c7c41e37927ba47faf639312fc936df3afad7926f5a httpd:2.4.39-alpine],SizeBytes:126894770,},ContainerImage{Names:[httpd@sha256:eb8ccf084cf3e80eece1add239effefd171eb39adbc154d33c14260d905d4060 httpd:2.4.38-alpine],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1],SizeBytes:121711221,},ContainerImage{Names:[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0],SizeBytes:71821984,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/sample-apiserver@sha256:1bafcc6fb1aa990b487850adba9cadc020e42d7905aa8a30481182a477ba24b0 gcr.io/kubernetes-e2e-test-images/sample-apiserver:1.10],SizeBytes:61365829,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727 gcr.io/kubernetes-e2e-test-images/agnhost:2.6],SizeBytes:57345321,},ContainerImage{Names:[quay.io/k8scsi/csi-provisioner@sha256:0efcb424f1dde9b9fb11a1a14f2e48ab47e1c3f08bc3a929990dcfcb1f7ab34f quay.io/k8scsi/csi-provisioner:v1.4.0-rc1],SizeBytes:54431016,},ContainerImage{Names:[quay.io/k8scsi/csi-snapshotter@sha256:e3d3e742e32d00488fdb401045b9b1d033d7ca0ab6e760f77b24750fc95e5f70 quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1],SizeBytes:51703561,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0],SizeBytes:46226754,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:0aba670b4d9d6b2e720bbf575d733156c676b693ca26501235444490300db838 quay.io/k8scsi/csi-attacher:v1.1.0],SizeBytes:42839085,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:7d46fb6eb8b890dc546029d1565d502b4a1d974d33625c6ee2bc7991b77fc1a1 quay.io/k8scsi/csi-resizer:v0.2.0],SizeBytes:42817100,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:f315c9042e56def3c05c6b04fe79ec9da6d39ddc557ca365a76cf35964ea08b6 quay.io/k8scsi/csi-resizer:v0.1.0],SizeBytes:42623056,},ContainerImage{Names:[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1],SizeBytes:42541759,},ContainerImage{Names:[gcr.io/google-containers/debian-base@sha256:6966a0aedd7592c18ff2dd803c08bd85780ee19f5e3a2e7cf908a4cd837afcde gcr.io/google-containers/debian-base:0.4.1],SizeBytes:42323657,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nonroot@sha256:d4ede5c74517090b6686219059118ed178cf4620f5db8781b32f806bb1e7395b gcr.io/kubernetes-e2e-test-images/nonroot:1.0],SizeBytes:42321438,},ContainerImage{Names:[redis@sha256:50899ea1ceed33fa03232f3ac57578a424faa1742c1ac9c7a7bdb95cdf19b858 redis:5.0.5-alpine],SizeBytes:29331594,},ContainerImage{Names:[quay.io/k8scsi/hostpathplugin@sha256:b4826e492fc1762fceaf9726f41575ca0a4567864d3d235da874818de18039de quay.io/k8scsi/hostpathplugin:v1.2.0-rc5],SizeBytes:28761497,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7],SizeBytes:22909487,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/echoserver@sha256:e9ba514b896cdf559eef8788b66c2c3ee55f3572df617647b4b0d8b6bf81cf19 gcr.io/kubernetes-e2e-test-images/echoserver:2.2],SizeBytes:21692741,},ContainerImage{Names:[quay.io/k8scsi/mock-driver@sha256:e0eed916b7d970bad2b7d9875f9ad16932f987f0f3d91ec5d86da68b0b5cc9d1 quay.io/k8scsi/mock-driver:v2.1.0],SizeBytes:16226335,},ContainerImage{Names:[nginx@sha256:485b610fefec7ff6c463ced9623314a04ed67e3945b9c08d7e53a47f6d108dc7 nginx:1.14-alpine],SizeBytes:16032814,},ContainerImage{Names:[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0],SizeBytes:15815995,},ContainerImage{Names:[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0],SizeBytes:14967303,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/dnsutils@sha256:2abeee84efb79c14d731966e034af33bf324d3b26ca28497555511ff094b3ddd gcr.io/kubernetes-e2e-test-images/dnsutils:1.1],SizeBytes:9349974,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nonewprivs@sha256:10066e9039219449fe3c81f38fe01928f87914150768ab81b62a468e51fa7411 gcr.io/kubernetes-e2e-test-images/nonewprivs:1.0],SizeBytes:6757579,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nautilus@sha256:33a732d4c42a266912a5091598a0f07653c9134db4b8d571690d8afd509e0bfc gcr.io/kubernetes-e2e-test-images/nautilus:1.0],SizeBytes:4753501,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/kitten@sha256:bcbc4875c982ab39aa7c4f6acf4a287f604e996d9f34a3fbda8c3d1a7457d1f6 gcr.io/kubernetes-e2e-test-images/kitten:1.0],SizeBytes:4747037,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/test-webserver@sha256:7f93d6e32798ff28bc6289254d0c2867fe2c849c8e46edc50f8624734309812e gcr.io/kubernetes-e2e-test-images/test-webserver:1.0],SizeBytes:4732240,},ContainerImage{Names:[alpine@sha256:8421d9a84432575381bfabd248f1eb56f3aa21d9d7cd2511583c68c9b7511d10 alpine:3.7],SizeBytes:4206494,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 gcr.io/kubernetes-e2e-test-images/mounttest:1.0],SizeBytes:1563521,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest-user@sha256:17319ca525ee003681fccf7e8c6b1b910ff4f49b653d939ac7f9b6e7c463933d gcr.io/kubernetes-e2e-test-images/mounttest-user:1.0],SizeBytes:1450451,},ContainerImage{Names:[busybox@sha256:1303dbf110c57f3edf68d9f5a16c082ec06c4cf7604831669faf2c712260b5a0 busybox:latest],SizeBytes:1219790,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea k8s.gcr.io/pause-amd64:3.1 k8s.gcr.io/pause:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 21:16:26.936: INFO: Logging kubelet events for node k8s-agentpool-24944505-vmss000000 Nov 13 21:16:26.991: INFO: Logging pods the kubelet thinks is on node k8s-agentpool-24944505-vmss000000 Nov 13 21:16:27.048: INFO: termination-message-container1e6b6341-f20c-4f16-935c-33a73d844cdb started at 2019-11-13 21:16:15 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.048: INFO: Container termination-message-container ready: false, restart count 0 Nov 13 21:16:27.048: INFO: kube-proxy-h9n4c started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.048: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 21:16:27.048: INFO: kubernetes-dashboard-65966766b9-lq4lm started at 2019-11-13 20:42:59 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.048: INFO: Container kubernetes-dashboard ready: true, restart count 0 Nov 13 21:16:27.048: INFO: pod-host-path-test started at 2019-11-13 21:15:51 +0000 UTC (0+2 container statuses recorded) Nov 13 21:16:27.048: INFO: Container test-container-1 ready: false, restart count 0 Nov 13 21:16:27.048: INFO: Container test-container-2 ready: false, restart count 0 Nov 13 21:16:27.048: INFO: hostexec-k8s-agentpool-24944505-vmss000000 started at 2019-11-13 21:14:24 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.048: INFO: Container agnhost ready: true, restart count 0 Nov 13 21:16:27.048: INFO: csi-snapshotter-0 started at 2019-11-13 21:10:12 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.048: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 13 21:16:27.048: INFO: pod-handle-http-request started at 2019-11-13 21:15:55 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.048: INFO: Container pod-handle-http-request ready: false, restart count 0 Nov 13 21:16:27.048: INFO: csi-hostpathplugin-0 started at <nil> (0+0 container statuses recorded) Nov 13 21:16:27.048: INFO: hostexec-k8s-agentpool-24944505-vmss000000 started at 2019-11-13 21:15:09 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.048: INFO: Container agnhost ready: true, restart count 0 Nov 13 21:16:27.048: INFO: security-context-5a6cd80c-9665-4a8d-abcb-ceb99fb0d26e started at 2019-11-13 21:15:58 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.048: INFO: Container write-pod ready: false, restart count 0 Nov 13 21:16:27.048: INFO: csi-hostpathplugin-0 started at 2019-11-13 21:13:27 +0000 UTC (0+3 container statuses recorded) Nov 13 21:16:27.048: INFO: Container hostpath ready: true, restart count 0 Nov 13 21:16:27.048: INFO: Container liveness-probe ready: true, restart count 0 Nov 13 21:16:27.048: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 13 21:16:27.048: INFO: metrics-server-58ff8c5ddf-r2d7f started at 2019-11-13 20:42:59 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.048: INFO: Container metrics-server ready: true, restart count 0 Nov 13 21:16:27.048: INFO: hostexec-k8s-agentpool-24944505-vmss000000 started at 2019-11-13 21:14:45 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.048: INFO: Container agnhost ready: true, restart count 0 Nov 13 21:16:27.048: INFO: azure-ip-masq-agent-qksp6 started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.048: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 21:16:27.048: INFO: tiller-deploy-7559b6b885-qxwkm started at 2019-11-13 20:42:59 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.048: INFO: Container tiller ready: true, restart count 0 Nov 13 21:16:27.048: INFO: sample-webhook-deployment-86d95b659d-h55rv started at 2019-11-13 21:15:17 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.048: INFO: Container sample-webhook ready: false, restart count 0 Nov 13 21:16:27.048: INFO: security-context-0500e38e-14b1-4091-bb6d-c80bcac61a8e started at 2019-11-13 21:15:45 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.048: INFO: Container write-pod ready: false, restart count 0 Nov 13 21:16:27.048: INFO: security-context-0b7d1ee2-13ae-4819-a020-a60a13614f93 started at 2019-11-13 21:16:00 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.048: INFO: Container write-pod ready: false, restart count 0 Nov 13 21:16:27.048: INFO: csi-hostpath-resizer-0 started at 2019-11-13 21:10:12 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.048: INFO: Container csi-resizer ready: true, restart count 0 Nov 13 21:16:27.048: INFO: security-context-cafb29ed-7842-4889-aa16-93e9dc6df626 started at 2019-11-13 21:14:53 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.048: INFO: Container write-pod ready: true, restart count 0 Nov 13 21:16:27.048: INFO: csi-hostpath-provisioner-0 started at 2019-11-13 21:15:02 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.048: INFO: Container csi-provisioner ready: true, restart count 0 Nov 13 21:16:27.048: INFO: hairpin started at 2019-11-13 21:16:16 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.048: INFO: Container agnhost ready: false, restart count 0 Nov 13 21:16:27.048: INFO: hostexec-k8s-agentpool-24944505-vmss000000 started at 2019-11-13 21:14:13 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.048: INFO: Container agnhost ready: true, restart count 0 Nov 13 21:16:27.048: INFO: security-context-1539320c-70ac-4929-b35d-8a3a8855b7ef started at 2019-11-13 21:14:52 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.048: INFO: Container write-pod ready: true, restart count 0 Nov 13 21:16:27.048: INFO: csi-snapshotter-0 started at 2019-11-13 21:15:05 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.048: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 13 21:16:27.048: INFO: csi-hostpath-provisioner-0 started at 2019-11-13 21:10:10 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.048: INFO: Container csi-provisioner ready: true, restart count 0 Nov 13 21:16:27.048: INFO: security-context-454a61dd-2a04-4e7a-91c4-9c89750923b3 started at 2019-11-13 21:15:46 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.048: INFO: Container write-pod ready: false, restart count 0 Nov 13 21:16:27.048: INFO: hostexec-k8s-agentpool-24944505-vmss000000 started at 2019-11-13 21:14:09 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.048: INFO: Container agnhost ready: true, restart count 0 Nov 13 21:16:27.048: INFO: pod-subpath-test-local-preprovisionedpv-6gch started at 2019-11-13 21:14:50 +0000 UTC (1+2 container statuses recorded) Nov 13 21:16:27.048: INFO: Init container test-init-subpath-local-preprovisionedpv-6gch ready: true, restart count 0 Nov 13 21:16:27.048: INFO: Container test-container-subpath-local-preprovisionedpv-6gch ready: false, restart count 0 Nov 13 21:16:27.048: INFO: Container test-container-volume-local-preprovisionedpv-6gch ready: false, restart count 0 Nov 13 21:16:27.048: INFO: csi-hostpath-attacher-0 started at 2019-11-13 21:14:59 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.048: INFO: Container csi-attacher ready: true, restart count 0 Nov 13 21:16:27.048: INFO: keyvault-flexvolume-sn47m started at 2019-11-13 20:42:59 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.048: INFO: Container keyvault-flexvolume ready: true, restart count 0 Nov 13 21:16:27.048: INFO: blobfuse-flexvol-installer-drhkj started at 2019-11-13 20:42:59 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.048: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 Nov 13 21:16:27.048: INFO: csi-hostpath-attacher-0 started at 2019-11-13 21:10:07 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.048: INFO: Container csi-attacher ready: true, restart count 0 Nov 13 21:16:27.048: INFO: csi-hostpath-resizer-0 started at 2019-11-13 21:15:04 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.049: INFO: Container csi-resizer ready: true, restart count 0 W1113 21:16:27.100876 92630 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 21:16:27.411: INFO: Latency metrics for node k8s-agentpool-24944505-vmss000000 Nov 13 21:16:27.411: INFO: Logging node info for node k8s-agentpool-24944505-vmss000001 Nov 13 21:16:27.473: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool-24944505-vmss000001 /api/v1/nodes/k8s-agentpool-24944505-vmss000001 da675d2e-0aad-4a8f-b94d-1311fb000563 51888 0 2019-11-13 20:42:24 +0000 UTC <nil> <nil> map[agentpool:agentpool beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool-24944505-vmss000001 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: storageprofile:managed storagetier:Premium_LRS] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-351":"k8s-agentpool-24944505-vmss000001","csi-hostpath-ephemeral-4187":"k8s-agentpool-24944505-vmss000001","csi-hostpath-ephemeral-5584":"k8s-agentpool-24944505-vmss000001","csi-hostpath-provisioning-1638":"k8s-agentpool-24944505-vmss000001","csi-hostpath-provisioning-225":"k8s-agentpool-24944505-vmss000001","csi-hostpath-provisioning-7099":"k8s-agentpool-24944505-vmss000001","csi-hostpath-volume-6291":"k8s-agentpool-24944505-vmss000001","csi-hostpath-volume-expand-7524":"k8s-agentpool-24944505-vmss000001","csi-hostpath-volume-expand-8458":"k8s-agentpool-24944505-vmss000001","csi-mock-csi-mock-volumes-2938":"csi-mock-csi-mock-volumes-2938","csi-mock-csi-mock-volumes-4469":"csi-mock-csi-mock-volumes-4469","csi-mock-csi-mock-volumes-4959":"csi-mock-csi-mock-volumes-4959","csi-mock-csi-mock-volumes-597":"csi-mock-csi-mock-volumes-597","csi-mock-csi-mock-volumes-8812":"csi-mock-csi-mock-volumes-8812","csi-mock-csi-mock-volumes-9088":"csi-mock-csi-mock-volumes-9088"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-24944505-vmss/virtualMachines/1,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16797569024 0} {<nil>} 16403876Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16011137024 0} {<nil>} 15635876Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 21:15:33 +0000 UTC,LastTransitionTime:2019-11-13 20:42:24 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 21:15:33 +0000 UTC,LastTransitionTime:2019-11-13 20:42:24 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 21:15:33 +0000 UTC,LastTransitionTime:2019-11-13 20:42:24 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 21:15:33 +0000 UTC,LastTransitionTime:2019-11-13 20:42:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.248.0.5,},NodeAddress{Type:Hostname,Address:k8s-agentpool-24944505-vmss000001,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:6ed41b2c4b884fccaf94748594531e06,SystemUUID:9CE5C2E3-7C34-C64A-9734-2477E19937B7,BootID:306ac3f6-a23b-423e-83c9-e3d5f3938f6c,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[quay.io/kubernetes_incubator/nfs-provisioner@sha256:df762117e3c891f2d2ddff46ecb0776ba1f9f3c44cfd7739b0683bcd7a7954a8 quay.io/kubernetes_incubator/nfs-provisioner:v2.2.2],SizeBytes:391772778,},ContainerImage{Names:[gluster/glusterdynamic-provisioner@sha256:90067cb05a7d217651e84576935934fd9dcff8e6bcdcbaa416bbf36fcd09dbd1 gluster/glusterdynamic-provisioner:v1.0],SizeBytes:373281573,},ContainerImage{Names:[gcr.io/google-samples/gb-frontend@sha256:35cb427341429fac3df10ff74600ea73e8ec0754d78f9ce89e0b4f3d70d53ba6 gcr.io/google-samples/gb-frontend:v6],SizeBytes:373099368,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/volume/nfs@sha256:c2ad734346f608a5f7d69cfded93c4e8094069320657bd372d12ba21dea3ea71 gcr.io/kubernetes-e2e-test-images/volume/nfs:1.0],SizeBytes:225358913,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/jessie-dnsutils@sha256:ad583e33cb284f7ef046673809b146ec4053cda19b54a85d2b180a86169715eb gcr.io/kubernetes-e2e-test-images/jessie-dnsutils:1.0],SizeBytes:195659796,},ContainerImage{Names:[httpd@sha256:addd70e4ee83f3bc9a4c1c7c41e37927ba47faf639312fc936df3afad7926f5a httpd:2.4.39-alpine],SizeBytes:126894770,},ContainerImage{Names:[httpd@sha256:eb8ccf084cf3e80eece1add239effefd171eb39adbc154d33c14260d905d4060 httpd:2.4.38-alpine],SizeBytes:123781643,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727 gcr.io/kubernetes-e2e-test-images/agnhost:2.6],SizeBytes:57345321,},ContainerImage{Names:[quay.io/k8scsi/csi-provisioner@sha256:0efcb424f1dde9b9fb11a1a14f2e48ab47e1c3f08bc3a929990dcfcb1f7ab34f quay.io/k8scsi/csi-provisioner:v1.4.0-rc1],SizeBytes:54431016,},ContainerImage{Names:[quay.io/k8scsi/csi-snapshotter@sha256:e3d3e742e32d00488fdb401045b9b1d033d7ca0ab6e760f77b24750fc95e5f70 quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1],SizeBytes:51703561,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0],SizeBytes:46226754,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:0aba670b4d9d6b2e720bbf575d733156c676b693ca26501235444490300db838 quay.io/k8scsi/csi-attacher:v1.1.0],SizeBytes:42839085,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:7d46fb6eb8b890dc546029d1565d502b4a1d974d33625c6ee2bc7991b77fc1a1 quay.io/k8scsi/csi-resizer:v0.2.0],SizeBytes:42817100,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:f315c9042e56def3c05c6b04fe79ec9da6d39ddc557ca365a76cf35964ea08b6 quay.io/k8scsi/csi-resizer:v0.1.0],SizeBytes:42623056,},ContainerImage{Names:[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0],SizeBytes:42488424,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nonroot@sha256:d4ede5c74517090b6686219059118ed178cf4620f5db8781b32f806bb1e7395b gcr.io/kubernetes-e2e-test-images/nonroot:1.0],SizeBytes:42321438,},ContainerImage{Names:[redis@sha256:50899ea1ceed33fa03232f3ac57578a424faa1742c1ac9c7a7bdb95cdf19b858 redis:5.0.5-alpine],SizeBytes:29331594,},ContainerImage{Names:[quay.io/k8scsi/hostpathplugin@sha256:b4826e492fc1762fceaf9726f41575ca0a4567864d3d235da874818de18039de quay.io/k8scsi/hostpathplugin:v1.2.0-rc5],SizeBytes:28761497,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7],SizeBytes:22909487,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/echoserver@sha256:e9ba514b896cdf559eef8788b66c2c3ee55f3572df617647b4b0d8b6bf81cf19 gcr.io/kubernetes-e2e-test-images/echoserver:2.2],SizeBytes:21692741,},ContainerImage{Names:[quay.io/k8scsi/mock-driver@sha256:e0eed916b7d970bad2b7d9875f9ad16932f987f0f3d91ec5d86da68b0b5cc9d1 quay.io/k8scsi/mock-driver:v2.1.0],SizeBytes:16226335,},ContainerImage{Names:[nginx@sha256:485b610fefec7ff6c463ced9623314a04ed67e3945b9c08d7e53a47f6d108dc7 nginx:1.14-alpine],SizeBytes:16032814,},ContainerImage{Names:[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0],SizeBytes:15815995,},ContainerImage{Names:[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0],SizeBytes:14967303,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/dnsutils@sha256:2abeee84efb79c14d731966e034af33bf324d3b26ca28497555511ff094b3ddd gcr.io/kubernetes-e2e-test-images/dnsutils:1.1],SizeBytes:9349974,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nautilus@sha256:33a732d4c42a266912a5091598a0f07653c9134db4b8d571690d8afd509e0bfc gcr.io/kubernetes-e2e-test-images/nautilus:1.0],SizeBytes:4753501,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/kitten@sha256:bcbc4875c982ab39aa7c4f6acf4a287f604e996d9f34a3fbda8c3d1a7457d1f6 gcr.io/kubernetes-e2e-test-images/kitten:1.0],SizeBytes:4747037,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/test-webserver@sha256:7f93d6e32798ff28bc6289254d0c2867fe2c849c8e46edc50f8624734309812e gcr.io/kubernetes-e2e-test-images/test-webserver:1.0],SizeBytes:4732240,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:4206620,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 gcr.io/kubernetes-e2e-test-images/mounttest:1.0],SizeBytes:1563521,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest-user@sha256:17319ca525ee003681fccf7e8c6b1b910ff4f49b653d939ac7f9b6e7c463933d gcr.io/kubernetes-e2e-test-images/mounttest-user:1.0],SizeBytes:1450451,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea k8s.gcr.io/pause-amd64:3.1 k8s.gcr.io/pause:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 21:16:27.474: INFO: Logging kubelet events for node k8s-agentpool-24944505-vmss000001 Nov 13 21:16:27.528: INFO: Logging pods the kubelet thinks is on node k8s-agentpool-24944505-vmss000001 Nov 13 21:16:27.589: INFO: pfpod started at 2019-11-13 21:14:01 +0000 UTC (0+2 container statuses recorded) Nov 13 21:16:27.589: INFO: Container portforwardtester ready: false, restart count 0 Nov 13 21:16:27.589: INFO: Container readiness ready: false, restart count 0 Nov 13 21:16:27.589: INFO: pod-subpath-test-local-preprovisionedpv-xgtl started at 2019-11-13 21:16:21 +0000 UTC (2+2 container statuses recorded) Nov 13 21:16:27.589: INFO: Init container init-volume-local-preprovisionedpv-xgtl ready: false, restart count 0 Nov 13 21:16:27.589: INFO: Init container test-init-subpath-local-preprovisionedpv-xgtl ready: false, restart count 0 Nov 13 21:16:27.589: INFO: Container test-container-subpath-local-preprovisionedpv-xgtl ready: false, restart count 0 Nov 13 21:16:27.589: INFO: Container test-container-volume-local-preprovisionedpv-xgtl ready: false, restart count 0 Nov 13 21:16:27.589: INFO: nfs-server started at 2019-11-13 21:12:32 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.589: INFO: Container nfs-server ready: true, restart count 0 Nov 13 21:16:27.589: INFO: blobfuse-flexvol-installer-cpm6h started at 2019-11-13 20:42:58 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.589: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 Nov 13 21:16:27.589: INFO: csi-mockplugin-0 started at 2019-11-13 21:15:00 +0000 UTC (0+3 container statuses recorded) Nov 13 21:16:27.589: INFO: Container csi-provisioner ready: true, restart count 0 Nov 13 21:16:27.589: INFO: Container driver-registrar ready: true, restart count 0 Nov 13 21:16:27.589: INFO: Container mock ready: true, restart count 0 Nov 13 21:16:27.589: INFO: pvc-volume-tester-c9p7v started at 2019-11-13 21:16:15 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.589: INFO: Container volume-tester ready: false, restart count 0 Nov 13 21:16:27.589: INFO: nfs-server started at 2019-11-13 21:10:25 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.589: INFO: Container nfs-server ready: true, restart count 0 Nov 13 21:16:27.589: INFO: hostexec-k8s-agentpool-24944505-vmss000001 started at 2019-11-13 21:11:25 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.589: INFO: Container agnhost ready: true, restart count 0 Nov 13 21:16:27.589: INFO: azure-ip-masq-agent-wwpg7 started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.589: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 21:16:27.589: INFO: pod-subpath-test-emptydir-gvcw started at 2019-11-13 21:16:11 +0000 UTC (2+2 container statuses recorded) Nov 13 21:16:27.589: INFO: Init container init-volume-emptydir-gvcw ready: false, restart count 0 Nov 13 21:16:27.589: INFO: Init container test-init-subpath-emptydir-gvcw ready: false, restart count 0 Nov 13 21:16:27.589: INFO: Container test-container-subpath-emptydir-gvcw ready: false, restart count 0 Nov 13 21:16:27.589: INFO: Container test-container-volume-emptydir-gvcw ready: false, restart count 0 Nov 13 21:16:27.589: INFO: hostexec-k8s-agentpool-24944505-vmss000001 started at 2019-11-13 21:15:01 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.589: INFO: Container agnhost ready: true, restart count 0 Nov 13 21:16:27.589: INFO: kube-proxy-7bpf7 started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.589: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 21:16:27.589: INFO: csi-hostpath-attacher-0 started at 2019-11-13 21:15:16 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.589: INFO: Container csi-attacher ready: true, restart count 0 Nov 13 21:16:27.589: INFO: nfs-client started at 2019-11-13 21:13:46 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.589: INFO: Container nfs-client ready: false, restart count 0 Nov 13 21:16:27.589: INFO: csi-hostpathplugin-0 started at 2019-11-13 21:15:21 +0000 UTC (0+3 container statuses recorded) Nov 13 21:16:27.589: INFO: Container hostpath ready: false, restart count 0 Nov 13 21:16:27.589: INFO: Container liveness-probe ready: false, restart count 0 Nov 13 21:16:27.589: INFO: Container node-driver-registrar ready: false, restart count 0 Nov 13 21:16:27.589: INFO: hostpath-injector started at 2019-11-13 21:16:25 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.589: INFO: Container hostpath-injector ready: false, restart count 0 Nov 13 21:16:27.589: INFO: hostexec-k8s-agentpool-24944505-vmss000001 started at 2019-11-13 21:15:33 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.589: INFO: Container agnhost ready: true, restart count 0 Nov 13 21:16:27.589: INFO: csi-hostpath-provisioner-0 started at 2019-11-13 21:15:22 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.589: INFO: Container csi-provisioner ready: false, restart count 0 Nov 13 21:16:27.589: INFO: pod-1 started at 2019-11-13 21:13:57 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.589: INFO: Container busybox ready: false, restart count 0 Nov 13 21:16:27.589: INFO: pod-init-1014ec29-f915-4785-beb0-c2e38197048b started at 2019-11-13 21:15:04 +0000 UTC (2+1 container statuses recorded) Nov 13 21:16:27.589: INFO: Init container init1 ready: true, restart count 0 Nov 13 21:16:27.589: INFO: Init container init2 ready: false, restart count 0 Nov 13 21:16:27.589: INFO: Container run1 ready: false, restart count 0 Nov 13 21:16:27.589: INFO: pod-0 started at 2019-11-13 21:13:55 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.589: INFO: Container busybox ready: false, restart count 0 Nov 13 21:16:27.589: INFO: pod-submit-remove-261e5ada-3b8a-4d79-ac63-dd17f887835a started at 2019-11-13 21:15:11 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.589: INFO: Container nginx ready: true, restart count 0 Nov 13 21:16:27.589: INFO: hostexec-k8s-agentpool-24944505-vmss000001 started at 2019-11-13 21:16:12 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.589: INFO: Container agnhost ready: false, restart count 0 Nov 13 21:16:27.589: INFO: pod-projected-configmaps-dd74a04e-ca68-4567-8fd3-62fc60587dc4 started at 2019-11-13 21:15:07 +0000 UTC (0+3 container statuses recorded) Nov 13 21:16:27.589: INFO: Container createcm-volume-test ready: true, restart count 0 Nov 13 21:16:27.589: INFO: Container delcm-volume-test ready: true, restart count 0 Nov 13 21:16:27.589: INFO: Container updcm-volume-test ready: true, restart count 0 Nov 13 21:16:27.589: INFO: csi-hostpath-resizer-0 started at 2019-11-13 21:15:22 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.589: INFO: Container csi-resizer ready: false, restart count 0 Nov 13 21:16:27.589: INFO: external-provisioner-scwrw started at 2019-11-13 21:09:09 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.589: INFO: Container nfs-provisioner ready: true, restart count 0 Nov 13 21:16:27.589: INFO: hostexec-k8s-agentpool-24944505-vmss000001 started at 2019-11-13 21:13:43 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.589: INFO: Container agnhost ready: true, restart count 0 Nov 13 21:16:27.589: INFO: hostexec-k8s-agentpool-24944505-vmss000001 started at 2019-11-13 21:15:15 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.589: INFO: Container agnhost ready: true, restart count 0 Nov 13 21:16:27.589: INFO: local-injector started at 2019-11-13 21:14:48 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.589: INFO: Container local-injector ready: true, restart count 0 Nov 13 21:16:27.589: INFO: pod-subpath-test-local-preprovisionedpv-8qzf started at 2019-11-13 21:16:25 +0000 UTC (1+2 container statuses recorded) Nov 13 21:16:27.589: INFO: Init container test-init-subpath-local-preprovisionedpv-8qzf ready: false, restart count 0 Nov 13 21:16:27.589: INFO: Container test-container-subpath-local-preprovisionedpv-8qzf ready: false, restart count 0 Nov 13 21:16:27.589: INFO: Container test-container-volume-local-preprovisionedpv-8qzf ready: false, restart count 0 Nov 13 21:16:27.589: INFO: sample-webhook-deployment-86d95b659d-2dvdp started at 2019-11-13 21:16:07 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.589: INFO: Container sample-webhook ready: false, restart count 0 Nov 13 21:16:27.589: INFO: pod-2 started at 2019-11-13 21:13:58 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.589: INFO: Container busybox ready: true, restart count 0 Nov 13 21:16:27.589: INFO: coredns-87f5d796-jd8bc started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.589: INFO: Container coredns ready: true, restart count 0 Nov 13 21:16:27.589: INFO: local-client started at 2019-11-13 21:14:48 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.589: INFO: Container local-client ready: true, restart count 0 Nov 13 21:16:27.589: INFO: csi-snapshotter-0 started at 2019-11-13 21:15:23 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.589: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 13 21:16:27.589: INFO: inline-volume-tester-qvfqv started at 2019-11-13 21:15:24 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.589: INFO: Container csi-volume-tester ready: false, restart count 0 Nov 13 21:16:27.589: INFO: keyvault-flexvolume-7phpr started at 2019-11-13 20:42:58 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:27.589: INFO: Container keyvault-flexvolume ready: true, restart count 0 W1113 21:16:27.642214 92630 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 21:16:28.137: INFO: Latency metrics for node k8s-agentpool-24944505-vmss000001 Nov 13 21:16:28.137: INFO: Logging node info for node k8s-master-24944505-vmss000000 Nov 13 21:16:28.188: INFO: Node Info: &Node{ObjectMeta:{k8s-master-24944505-vmss000000 /api/v1/nodes/k8s-master-24944505-vmss000000 048fbae2-b97f-43a2-b471-398180136f12 51641 0 2019-11-13 20:42:13 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-24944505-vmss000000 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-24944505-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284887552 0} {<nil>} 7114148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6498455552 0} {<nil>} 6346148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 21:16:08 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 21:16:08 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 21:16:08 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 21:16:08 +0000 UTC,LastTransitionTime:2019-11-13 20:42:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.4,},NodeAddress{Type:Hostname,Address:k8s-master-24944505-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:7bd754c0dc8647518c3043bcae3a76cc,SystemUUID:FA5CFE99-8148-6C46-B4E6-5EA2A5DE9D2A,BootID:74f7b0a2-e85c-438e-a642-c5b32900bb28,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:3237a53b135d8692ade82eca845fd0259ecbfcbe05fad35400e7d5c2018c4abd k8sprow.azurecr.io/azure-cloud-controller-manager:1194710255657816064-e688ecb0],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 21:16:28.188: INFO: Logging kubelet events for node k8s-master-24944505-vmss000000 Nov 13 21:16:28.248: INFO: Logging pods the kubelet thinks is on node k8s-master-24944505-vmss000000 Nov 13 21:16:28.335: INFO: kube-scheduler-k8s-master-24944505-vmss000000 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:28.335: INFO: Container kube-scheduler ready: true, restart count 0 Nov 13 21:16:28.335: INFO: azure-ip-masq-agent-6pnst started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:28.335: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 21:16:28.335: INFO: kube-proxy-d9jjh started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:28.335: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 21:16:28.335: INFO: cloud-controller-manager-k8s-master-24944505-vmss000000 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:28.335: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 13 21:16:28.335: INFO: kube-addon-manager-k8s-master-24944505-vmss000000 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:28.335: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 13 21:16:28.335: INFO: kube-apiserver-k8s-master-24944505-vmss000000 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:28.335: INFO: Container kube-apiserver ready: true, restart count 0 Nov 13 21:16:28.335: INFO: kube-controller-manager-k8s-master-24944505-vmss000000 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:28.335: INFO: Container kube-controller-manager ready: true, restart count 0 W1113 21:16:28.387266 92630 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 21:16:28.502: INFO: Latency metrics for node k8s-master-24944505-vmss000000 Nov 13 21:16:28.502: INFO: Logging node info for node k8s-master-24944505-vmss000001 Nov 13 21:16:28.553: INFO: Node Info: &Node{ObjectMeta:{k8s-master-24944505-vmss000001 /api/v1/nodes/k8s-master-24944505-vmss000001 a2619878-b119-44f9-b89f-4073bebbb56c 51548 0 2019-11-13 20:42:40 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-24944505-vmss000001 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.6.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-24944505-vmss/virtualMachines/1,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.6.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284883456 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6498451456 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:43:08 +0000 UTC,LastTransitionTime:2019-11-13 20:43:08 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 21:16:05 +0000 UTC,LastTransitionTime:2019-11-13 20:42:34 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 21:16:05 +0000 UTC,LastTransitionTime:2019-11-13 20:42:34 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 21:16:05 +0000 UTC,LastTransitionTime:2019-11-13 20:42:34 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 21:16:05 +0000 UTC,LastTransitionTime:2019-11-13 20:42:40 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.5,},NodeAddress{Type:Hostname,Address:k8s-master-24944505-vmss000001,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:c866a7c1ed64481990416e24387dd317,SystemUUID:EBE64557-2AC8-5740-9FFE-AB9F12412FBC,BootID:3dbb0e0e-0479-4821-bc3f-a8269d250e4c,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:3237a53b135d8692ade82eca845fd0259ecbfcbe05fad35400e7d5c2018c4abd k8sprow.azurecr.io/azure-cloud-controller-manager:1194710255657816064-e688ecb0],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 21:16:28.553: INFO: Logging kubelet events for node k8s-master-24944505-vmss000001 Nov 13 21:16:28.607: INFO: Logging pods the kubelet thinks is on node k8s-master-24944505-vmss000001 Nov 13 21:16:28.680: INFO: kube-apiserver-k8s-master-24944505-vmss000001 started at 2019-11-13 20:42:28 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:28.680: INFO: Container kube-apiserver ready: true, restart count 0 Nov 13 21:16:28.680: INFO: kube-controller-manager-k8s-master-24944505-vmss000001 started at 2019-11-13 20:42:28 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:28.681: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 13 21:16:28.681: INFO: kube-proxy-w4bz5 started at 2019-11-13 20:42:41 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:28.681: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 21:16:28.681: INFO: azure-ip-masq-agent-bs52d started at 2019-11-13 20:42:41 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:28.681: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 21:16:28.681: INFO: kube-scheduler-k8s-master-24944505-vmss000001 started at 2019-11-13 20:42:28 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:28.681: INFO: Container kube-scheduler ready: true, restart count 0 Nov 13 21:16:28.681: INFO: cloud-controller-manager-k8s-master-24944505-vmss000001 started at 2019-11-13 20:42:28 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:28.681: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 13 21:16:28.681: INFO: kube-addon-manager-k8s-master-24944505-vmss000001 started at 2019-11-13 20:42:28 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:28.681: INFO: Container kube-addon-manager ready: true, restart count 0 W1113 21:16:28.733476 92630 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 21:16:28.859: INFO: Latency metrics for node k8s-master-24944505-vmss000001 Nov 13 21:16:28.859: INFO: Logging node info for node k8s-master-24944505-vmss000002 Nov 13 21:16:28.910: INFO: Node Info: &Node{ObjectMeta:{k8s-master-24944505-vmss000002 /api/v1/nodes/k8s-master-24944505-vmss000002 f3b05267-a009-4dba-9f97-4f772e48707e 51637 0 2019-11-13 20:42:13 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-24944505-vmss000002 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.4.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-24944505-vmss/virtualMachines/2,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284887552 0} {<nil>} 7114148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6498455552 0} {<nil>} 6346148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 21:16:08 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 21:16:08 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 21:16:08 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 21:16:08 +0000 UTC,LastTransitionTime:2019-11-13 20:42:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.6,},NodeAddress{Type:Hostname,Address:k8s-master-24944505-vmss000002,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:fc1777f37ced4b9c97acb9b30e276c4f,SystemUUID:E9DF19DC-D02C-714C-A7D6-0BA6CE959C55,BootID:622958a8-d45d-4edf-9266-4dbb759e2359,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:3237a53b135d8692ade82eca845fd0259ecbfcbe05fad35400e7d5c2018c4abd k8sprow.azurecr.io/azure-cloud-controller-manager:1194710255657816064-e688ecb0],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 21:16:28.910: INFO: Logging kubelet events for node k8s-master-24944505-vmss000002 Nov 13 21:16:28.965: INFO: Logging pods the kubelet thinks is on node k8s-master-24944505-vmss000002 Nov 13 21:16:29.034: INFO: kube-apiserver-k8s-master-24944505-vmss000002 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:29.034: INFO: Container kube-apiserver ready: true, restart count 0 Nov 13 21:16:29.034: INFO: kube-controller-manager-k8s-master-24944505-vmss000002 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:29.034: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 13 21:16:29.034: INFO: kube-scheduler-k8s-master-24944505-vmss000002 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:29.034: INFO: Container kube-scheduler ready: true, restart count 0 Nov 13 21:16:29.034: INFO: cloud-controller-manager-k8s-master-24944505-vmss000002 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:29.034: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 13 21:16:29.034: INFO: azure-ip-masq-agent-6skf2 started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:29.034: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 21:16:29.034: INFO: kube-proxy-xwz78 started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:29.034: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 21:16:29.034: INFO: kube-addon-manager-k8s-master-24944505-vmss000002 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:29.034: INFO: Container kube-addon-manager ready: true, restart count 0 W1113 21:16:29.088655 92630 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 21:16:29.206: INFO: Latency metrics for node k8s-master-24944505-vmss000002 Nov 13 21:16:29.206: INFO: Logging node info for node k8s-master-24944505-vmss000003 Nov 13 21:16:29.258: INFO: Node Info: &Node{ObjectMeta:{k8s-master-24944505-vmss000003 /api/v1/nodes/k8s-master-24944505-vmss000003 888cc19e-c62d-4945-a425-97d14e38a0b8 51468 0 2019-11-13 20:42:35 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-24944505-vmss000003 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.5.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-24944505-vmss/virtualMachines/3,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.5.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284887552 0} {<nil>} 7114148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6498455552 0} {<nil>} 6346148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 21:16:00 +0000 UTC,LastTransitionTime:2019-11-13 20:42:29 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 21:16:00 +0000 UTC,LastTransitionTime:2019-11-13 20:42:29 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 21:16:00 +0000 UTC,LastTransitionTime:2019-11-13 20:42:29 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 21:16:00 +0000 UTC,LastTransitionTime:2019-11-13 20:42:35 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.7,},NodeAddress{Type:Hostname,Address:k8s-master-24944505-vmss000003,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:2da5fe5a442b4e178a2b62ba91366e4b,SystemUUID:D2D1E816-D70E-3D40-B959-B00A2B76A83A,BootID:c5929aa1-3dac-45d5-bb2c-174c73c7300e,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:3237a53b135d8692ade82eca845fd0259ecbfcbe05fad35400e7d5c2018c4abd k8sprow.azurecr.io/azure-cloud-controller-manager:1194710255657816064-e688ecb0],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 21:16:29.258: INFO: Logging kubelet events for node k8s-master-24944505-vmss000003 Nov 13 21:16:29.313: INFO: Logging pods the kubelet thinks is on node k8s-master-24944505-vmss000003 Nov 13 21:16:29.396: INFO: kube-addon-manager-k8s-master-24944505-vmss000003 started at 2019-11-13 20:42:22 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:29.396: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 13 21:16:29.396: INFO: kube-apiserver-k8s-master-24944505-vmss000003 started at 2019-11-13 20:42:23 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:29.396: INFO: Container kube-apiserver ready: true, restart count 0 Nov 13 21:16:29.396: INFO: kube-controller-manager-k8s-master-24944505-vmss000003 started at 2019-11-13 20:42:23 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:29.396: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 13 21:16:29.396: INFO: kube-scheduler-k8s-master-24944505-vmss000003 started at 2019-11-13 20:42:23 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:29.396: INFO: Container kube-scheduler ready: true, restart count 0 Nov 13 21:16:29.396: INFO: cloud-controller-manager-k8s-master-24944505-vmss000003 started at 2019-11-13 20:42:44 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:29.396: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 13 21:16:29.396: INFO: azure-ip-masq-agent-nlqss started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:29.396: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 21:16:29.396: INFO: kube-proxy-kgh5h started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:29.396: INFO: Container kube-proxy ready: true, restart count 0 W1113 21:16:29.452935 92630 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 21:16:29.567: INFO: Latency metrics for node k8s-master-24944505-vmss000003 Nov 13 21:16:29.567: INFO: Logging node info for node k8s-master-24944505-vmss000004 Nov 13 21:16:29.624: INFO: Node Info: &Node{ObjectMeta:{k8s-master-24944505-vmss000004 /api/v1/nodes/k8s-master-24944505-vmss000004 a26d844f-0612-406b-9a65-7e2ac952a2d5 51640 0 2019-11-13 20:42:14 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-24944505-vmss000004 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-24944505-vmss/virtualMachines/4,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284887552 0} {<nil>} 7114148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6498455552 0} {<nil>} 6346148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 21:16:08 +0000 UTC,LastTransitionTime:2019-11-13 20:42:09 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 21:16:08 +0000 UTC,LastTransitionTime:2019-11-13 20:42:09 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 21:16:08 +0000 UTC,LastTransitionTime:2019-11-13 20:42:09 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 21:16:08 +0000 UTC,LastTransitionTime:2019-11-13 20:42:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.8,},NodeAddress{Type:Hostname,Address:k8s-master-24944505-vmss000004,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:2208959aca1045d9b48fce8d784c1e41,SystemUUID:E253DB3A-3A4D-5648-8169-B2E2976CFB4B,BootID:09bc6101-2da9-4c8e-8274-43db8c5fe1cf,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:3237a53b135d8692ade82eca845fd0259ecbfcbe05fad35400e7d5c2018c4abd k8sprow.azurecr.io/azure-cloud-controller-manager:1194710255657816064-e688ecb0],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 21:16:29.626: INFO: Logging kubelet events for node k8s-master-24944505-vmss000004 Nov 13 21:16:29.683: INFO: Logging pods the kubelet thinks is on node k8s-master-24944505-vmss000004 Nov 13 21:16:29.756: INFO: kube-addon-manager-k8s-master-24944505-vmss000004 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:29.756: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 13 21:16:29.756: INFO: kube-apiserver-k8s-master-24944505-vmss000004 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:29.756: INFO: Container kube-apiserver ready: true, restart count 0 Nov 13 21:16:29.756: INFO: kube-controller-manager-k8s-master-24944505-vmss000004 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:29.756: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 13 21:16:29.757: INFO: kube-scheduler-k8s-master-24944505-vmss000004 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:29.757: INFO: Container kube-scheduler ready: true, restart count 0 Nov 13 21:16:29.757: INFO: azure-ip-masq-agent-xnw2h started at 2019-11-13 20:42:38 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:29.757: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 21:16:29.757: INFO: kube-proxy-fld6p started at 2019-11-13 20:42:38 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:29.757: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 21:16:29.757: INFO: cloud-controller-manager-k8s-master-24944505-vmss000004 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:16:29.757: INFO: Container cloud-controller-manager ready: true, restart count 0 W1113 21:16:29.809800 92630 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 21:16:29.929: INFO: Latency metrics for node k8s-master-24944505-vmss000004 Nov 13 21:16:29.929: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "pods-2091" for this suite. Nov 13 21:16:58.146: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered Nov 13 21:16:59.864: INFO: namespace pods-2091 deletion completed in 29.881420284s
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[sig\-cli\]\sKubectl\sPort\sforwarding\sWith\sa\sserver\slistening\son\slocalhost\sthat\sexpects\sNO\sclient\srequest\sshould\ssupport\sa\sclient\sthat\sconnects\,\ssends\sDATA\,\sand\sdisconnects$'
test/e2e/kubectl/portforward.go:474 Nov 13 21:28:01.372: Container did not terminate: Gave up after waiting 5m0s for pod "pfpod" to be "container terminated" test/e2e/kubectl/portforward.go:242from junit_25.xml
[BeforeEach] [sig-cli] Kubectl Port forwarding test/e2e/framework/framework.go:151 �[1mSTEP�[0m: Creating a kubernetes client Nov 13 21:20:08.769: INFO: >>> kubeConfig: /workspace/aks880331199/kubeconfig/kubeconfig.westus2.json �[1mSTEP�[0m: Building a namespace api object, basename port-forwarding �[1mSTEP�[0m: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in port-forwarding-905 �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace [It] should support a client that connects, sends DATA, and disconnects test/e2e/kubectl/portforward.go:474 �[1mSTEP�[0m: Creating the target pod �[1mSTEP�[0m: Running 'kubectl port-forward' Nov 13 21:22:57.387: INFO: starting port-forward command and streaming output Nov 13 21:22:57.387: INFO: Asynchronously running '/usr/local/bin/kubectl kubectl --server=https://kubetest-74ff5bf7-0652-11ea-bade-36185e316941.westus2.cloudapp.azure.com --kubeconfig=/workspace/aks880331199/kubeconfig/kubeconfig.westus2.json port-forward --namespace=port-forwarding-905 pfpod :80' Nov 13 21:22:57.387: INFO: reading from `kubectl port-forward` command's stdout �[1mSTEP�[0m: Dialing the local port �[1mSTEP�[0m: Reading data from the local port �[1mSTEP�[0m: Waiting for the target pod to stop running Nov 13 21:22:59.593: INFO: Waiting up to 5m0s for pod "pfpod" in namespace "port-forwarding-905" to be "container terminated" Nov 13 21:22:59.646: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=true. Elapsed: 53.051289ms Nov 13 21:23:01.699: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=true. Elapsed: 2.10576586s Nov 13 21:23:03.751: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 4.157905976s Nov 13 21:23:05.803: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 6.209717633s Nov 13 21:23:07.855: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 8.262003981s Nov 13 21:23:09.908: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 10.314572384s Nov 13 21:23:11.959: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 12.366182815s Nov 13 21:23:14.012: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 14.418782962s Nov 13 21:23:16.065: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 16.472096357s Nov 13 21:23:18.121: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 18.527770549s Nov 13 21:23:20.174: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 20.580676029s Nov 13 21:23:22.226: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 22.633330966s Nov 13 21:23:24.278: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 24.685469859s Nov 13 21:23:26.331: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 26.73792157s Nov 13 21:23:28.383: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 28.789976835s Nov 13 21:23:30.436: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 30.842691662s Nov 13 21:23:32.494: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 32.900802691s Nov 13 21:23:34.547: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 34.953729698s Nov 13 21:23:36.600: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 37.006795488s Nov 13 21:23:38.652: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 39.059458065s Nov 13 21:23:40.705: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 41.111845552s Nov 13 21:23:42.757: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 43.164231508s Nov 13 21:23:44.810: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 45.216929574s Nov 13 21:23:46.862: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 47.269313232s Nov 13 21:23:48.915: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 49.322368129s Nov 13 21:23:50.976: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 51.382785097s Nov 13 21:23:53.028: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 53.43489879s Nov 13 21:23:55.080: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 55.487326377s Nov 13 21:23:57.133: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 57.5398805s Nov 13 21:23:59.186: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 59.592946498s Nov 13 21:24:01.242: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 1m1.649393863s Nov 13 21:24:03.295: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 1m3.702157232s Nov 13 21:24:05.352: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 1m5.759238779s Nov 13 21:24:07.405: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 1m7.811789192s Nov 13 21:24:09.457: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 1m9.864332368s Nov 13 21:24:11.510: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 1m11.916755129s Nov 13 21:24:13.562: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 1m13.968960689s Nov 13 21:24:15.615: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 1m16.021946093s Nov 13 21:24:17.671: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 1m18.07811839s Nov 13 21:24:19.724: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 1m20.130986529s Nov 13 21:24:21.776: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 1m22.182761404s Nov 13 21:24:23.828: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 1m24.234847204s Nov 13 21:24:25.881: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 1m26.288032949s Nov 13 21:24:27.933: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 1m28.339883692s Nov 13 21:24:29.985: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 1m30.392016102s Nov 13 21:24:32.037: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 1m32.443871033s Nov 13 21:24:34.089: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 1m34.496370724s Nov 13 21:24:36.142: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 1m36.548971988s Nov 13 21:24:38.195: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 1m38.601641898s Nov 13 21:24:40.250: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 1m40.656921738s Nov 13 21:24:42.303: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 1m42.709961498s Nov 13 21:24:44.356: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 1m44.762904094s Nov 13 21:24:46.411: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 1m46.818259891s Nov 13 21:24:48.464: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 1m48.870884725s Nov 13 21:24:50.516: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 1m50.922949365s Nov 13 21:24:52.569: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 1m52.975578429s Nov 13 21:24:54.621: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 1m55.028512725s Nov 13 21:24:56.674: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 1m57.081063145s Nov 13 21:24:58.726: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 1m59.132921372s Nov 13 21:25:00.778: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 2m1.185403041s Nov 13 21:25:02.831: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 2m3.238051377s Nov 13 21:25:04.885: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 2m5.29242952s Nov 13 21:25:06.938: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 2m7.345426067s Nov 13 21:25:08.992: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 2m9.399389556s Nov 13 21:25:11.045: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 2m11.452039154s Nov 13 21:25:13.099: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 2m13.505693167s Nov 13 21:25:15.151: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 2m15.558370249s Nov 13 21:25:17.205: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 2m17.612470032s Nov 13 21:25:19.258: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 2m19.665285204s Nov 13 21:25:21.311: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 2m21.717820042s Nov 13 21:25:23.363: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 2m23.770325154s Nov 13 21:25:25.416: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 2m25.822848477s Nov 13 21:25:27.468: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 2m27.8748998s Nov 13 21:25:29.521: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 2m29.927766629s Nov 13 21:25:31.573: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 2m31.979922621s Nov 13 21:25:33.627: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 2m34.03383097s Nov 13 21:25:35.679: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 2m36.085938446s Nov 13 21:25:37.731: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 2m38.138394048s Nov 13 21:25:39.783: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 2m40.190564398s Nov 13 21:25:41.836: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 2m42.242903651s Nov 13 21:25:43.889: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 2m44.295977309s Nov 13 21:25:45.941: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 2m46.347811138s Nov 13 21:25:47.993: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 2m48.400123877s Nov 13 21:25:50.046: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 2m50.45277122s Nov 13 21:25:52.099: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 2m52.505766439s Nov 13 21:25:54.151: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 2m54.558098105s Nov 13 21:25:56.203: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 2m56.610389958s Nov 13 21:25:58.256: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 2m58.662962097s Nov 13 21:26:00.309: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 3m0.715979145s Nov 13 21:26:02.361: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 3m2.76803215s Nov 13 21:26:04.414: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 3m4.820962625s Nov 13 21:26:06.469: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 3m6.876098137s Nov 13 21:26:08.521: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 3m8.927983686s Nov 13 21:26:10.573: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 3m10.980448373s Nov 13 21:26:12.626: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 3m13.032838216s Nov 13 21:26:14.679: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 3m15.085977139s Nov 13 21:26:16.732: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 3m17.139358672s Nov 13 21:26:18.785: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 3m19.191877371s Nov 13 21:26:20.838: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 3m21.244612464s Nov 13 21:26:22.890: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 3m23.297206801s Nov 13 21:26:24.942: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 3m25.349239928s Nov 13 21:26:26.995: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 3m27.401853657s Nov 13 21:26:29.048: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 3m29.454636523s Nov 13 21:26:31.100: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 3m31.506874932s Nov 13 21:26:33.157: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 3m33.563733004s Nov 13 21:26:35.210: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 3m35.617188128s Nov 13 21:26:37.263: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 3m37.669594464s Nov 13 21:26:39.315: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 3m39.721847048s Nov 13 21:26:41.367: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 3m41.773951715s Nov 13 21:26:43.420: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 3m43.826759914s Nov 13 21:26:45.472: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 3m45.878764404s Nov 13 21:26:47.523: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 3m47.930454547s Nov 13 21:26:49.576: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 3m49.982625165s Nov 13 21:26:51.633: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 3m52.040001324s Nov 13 21:26:53.689: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 3m54.095630808s Nov 13 21:26:55.740: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 3m56.147165567s Nov 13 21:26:57.793: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 3m58.199950928s Nov 13 21:26:59.846: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 4m0.253304723s Nov 13 21:27:01.898: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 4m2.305521526s Nov 13 21:27:03.951: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 4m4.357974316s Nov 13 21:27:06.003: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 4m6.410557905s Nov 13 21:27:08.056: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 4m8.462524214s Nov 13 21:27:10.109: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 4m10.515834769s Nov 13 21:27:12.162: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 4m12.568628035s Nov 13 21:27:14.214: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 4m14.620679885s Nov 13 21:27:16.265: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 4m16.67242988s Nov 13 21:27:18.318: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 4m18.725465074s Nov 13 21:27:20.374: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 4m20.781317792s Nov 13 21:27:22.428: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 4m22.834753153s Nov 13 21:27:24.480: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 4m24.887135681s Nov 13 21:27:26.532: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 4m26.93906826s Nov 13 21:27:28.584: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 4m28.991273937s Nov 13 21:27:30.637: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 4m31.043727003s Nov 13 21:27:32.690: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 4m33.097324505s Nov 13 21:27:34.743: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 4m35.149880756s Nov 13 21:27:36.795: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 4m37.201947757s Nov 13 21:27:38.847: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 4m39.253964002s Nov 13 21:27:40.900: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 4m41.306815983s Nov 13 21:27:42.952: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 4m43.358835963s Nov 13 21:27:45.004: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 4m45.410960487s Nov 13 21:27:47.057: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 4m47.464200856s Nov 13 21:27:49.110: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 4m49.516675056s Nov 13 21:27:51.161: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 4m51.568546815s Nov 13 21:27:53.214: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 4m53.62099331s Nov 13 21:27:55.266: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 4m55.67342642s Nov 13 21:27:57.319: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 4m57.726429274s Nov 13 21:27:59.372: INFO: Pod "pfpod": Phase="Running", Reason="", readiness=false. Elapsed: 4m59.778919524s Nov 13 21:28:01.372: FAIL: Container did not terminate: Gave up after waiting 5m0s for pod "pfpod" to be "container terminated" �[1mSTEP�[0m: Closing the connection to the local port [AfterEach] [sig-cli] Kubectl Port forwarding test/e2e/framework/framework.go:152 �[1mSTEP�[0m: Collecting events from namespace "port-forwarding-905". �[1mSTEP�[0m: Found 11 events. Nov 13 21:28:01.425: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for pfpod: {default-scheduler } Scheduled: Successfully assigned port-forwarding-905/pfpod to k8s-agentpool-24944505-vmss000001 Nov 13 21:28:01.425: INFO: At 2019-11-13 21:20:51 +0000 UTC - event for pfpod: {kubelet k8s-agentpool-24944505-vmss000001} Pulling: Pulling image "gcr.io/kubernetes-e2e-test-images/agnhost:2.6" Nov 13 21:28:01.425: INFO: At 2019-11-13 21:20:52 +0000 UTC - event for pfpod: {kubelet k8s-agentpool-24944505-vmss000001} Pulled: Successfully pulled image "gcr.io/kubernetes-e2e-test-images/agnhost:2.6" Nov 13 21:28:01.425: INFO: At 2019-11-13 21:20:59 +0000 UTC - event for pfpod: {kubelet k8s-agentpool-24944505-vmss000001} Created: Created container readiness Nov 13 21:28:01.425: INFO: At 2019-11-13 21:21:03 +0000 UTC - event for pfpod: {kubelet k8s-agentpool-24944505-vmss000001} Started: Started container readiness Nov 13 21:28:01.425: INFO: At 2019-11-13 21:21:03 +0000 UTC - event for pfpod: {kubelet k8s-agentpool-24944505-vmss000001} Pulling: Pulling image "gcr.io/kubernetes-e2e-test-images/agnhost:2.6" Nov 13 21:28:01.425: INFO: At 2019-11-13 21:21:03 +0000 UTC - event for pfpod: {kubelet k8s-agentpool-24944505-vmss000001} Pulled: Successfully pulled image "gcr.io/kubernetes-e2e-test-images/agnhost:2.6" Nov 13 21:28:01.425: INFO: At 2019-11-13 21:21:06 +0000 UTC - event for pfpod: {kubelet k8s-agentpool-24944505-vmss000001} Created: Created container portforwardtester Nov 13 21:28:01.425: INFO: At 2019-11-13 21:21:07 +0000 UTC - event for pfpod: {kubelet k8s-agentpool-24944505-vmss000001} Started: Started container portforwardtester Nov 13 21:28:01.425: INFO: At 2019-11-13 21:23:00 +0000 UTC - event for pfpod: {kubelet k8s-agentpool-24944505-vmss000001} Unhealthy: Readiness probe failed: Nov 13 21:28:01.425: INFO: At 2019-11-13 21:27:07 +0000 UTC - event for pfpod: {taint-controller } TaintManagerEviction: Cancelling deletion of Pod port-forwarding-905/pfpod Nov 13 21:28:01.478: INFO: POD NODE PHASE GRACE CONDITIONS Nov 13 21:28:01.478: INFO: pfpod k8s-agentpool-24944505-vmss000001 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:20:18 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:23:02 +0000 UTC ContainersNotReady containers with unready status: [readiness]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:23:02 +0000 UTC ContainersNotReady containers with unready status: [readiness]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:20:09 +0000 UTC }] Nov 13 21:28:01.478: INFO: Nov 13 21:28:01.632: INFO: Logging node info for node k8s-agentpool-24944505-vmss000000 Nov 13 21:28:01.684: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool-24944505-vmss000000 /api/v1/nodes/k8s-agentpool-24944505-vmss000000 5bc7cf9e-80da-4e26-a2c2-20757e870d68 59386 0 2019-11-13 20:42:13 +0000 UTC <nil> <nil> map[agentpool:agentpool beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool-24944505-vmss000000 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: storageprofile:managed storagetier:Premium_LRS] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-1137":"k8s-agentpool-24944505-vmss000000","csi-hostpath-ephemeral-9135":"k8s-agentpool-24944505-vmss000000","csi-hostpath-provisioning-2585":"k8s-agentpool-24944505-vmss000000","csi-hostpath-provisioning-3303":"k8s-agentpool-24944505-vmss000000","csi-hostpath-provisioning-7066":"k8s-agentpool-24944505-vmss000000","csi-hostpath-provisioning-716":"k8s-agentpool-24944505-vmss000000","csi-hostpath-provisioning-833":"k8s-agentpool-24944505-vmss000000","csi-hostpath-volume-9286":"k8s-agentpool-24944505-vmss000000","csi-hostpath-volume-expand-171":"k8s-agentpool-24944505-vmss000000","csi-hostpath-volume-expand-4114":"k8s-agentpool-24944505-vmss000000","csi-hostpath-volume-expand-4561":"k8s-agentpool-24944505-vmss000000","csi-hostpath-volume-expand-9975":"k8s-agentpool-24944505-vmss000000","csi-mock-csi-mock-volumes-2863":"csi-mock-csi-mock-volumes-2863","csi-mock-csi-mock-volumes-5360":"csi-mock-csi-mock-volumes-5360","csi-mock-csi-mock-volumes-6481":"csi-mock-csi-mock-volumes-6481","csi-mock-csi-mock-volumes-6951":"csi-mock-csi-mock-volumes-6951","csi-mock-csi-mock-volumes-8439":"csi-mock-csi-mock-volumes-8439","csi-mock-csi-mock-volumes-8606":"csi-mock-csi-mock-volumes-8606"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-24944505-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16797569024 0} {<nil>} 16403876Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16011137024 0} {<nil>} 15635876Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 21:27:42 +0000 UTC,LastTransitionTime:2019-11-13 20:42:03 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 21:27:42 +0000 UTC,LastTransitionTime:2019-11-13 20:42:03 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 21:27:42 +0000 UTC,LastTransitionTime:2019-11-13 20:42:03 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 21:27:42 +0000 UTC,LastTransitionTime:2019-11-13 20:42:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.248.0.4,},NodeAddress{Type:Hostname,Address:k8s-agentpool-24944505-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:0a2e8938ebfc440180ed5c0b5a9edbe7,SystemUUID:8075A1F3-D764-7248-9570-A764E51D7FF2,BootID:4f2b3cbe-c61c-4c6c-93ab-18fca0df5bc5,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[quay.io/kubernetes_incubator/nfs-provisioner@sha256:df762117e3c891f2d2ddff46ecb0776ba1f9f3c44cfd7739b0683bcd7a7954a8 quay.io/kubernetes_incubator/nfs-provisioner:v2.2.2],SizeBytes:391772778,},ContainerImage{Names:[gcr.io/google-samples/gb-frontend@sha256:35cb427341429fac3df10ff74600ea73e8ec0754d78f9ce89e0b4f3d70d53ba6 gcr.io/google-samples/gb-frontend:v6],SizeBytes:373099368,},ContainerImage{Names:[k8s.gcr.io/etcd@sha256:12c2c5e5731c3bcd56e6f1c05c0f9198b6f06793fa7fca2fb43aab9622dc4afa k8s.gcr.io/etcd:3.3.15],SizeBytes:246640776,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/volume/nfs@sha256:c2ad734346f608a5f7d69cfded93c4e8094069320657bd372d12ba21dea3ea71 gcr.io/kubernetes-e2e-test-images/volume/nfs:1.0],SizeBytes:225358913,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/jessie-dnsutils@sha256:ad583e33cb284f7ef046673809b146ec4053cda19b54a85d2b180a86169715eb gcr.io/kubernetes-e2e-test-images/jessie-dnsutils:1.0],SizeBytes:195659796,},ContainerImage{Names:[httpd@sha256:addd70e4ee83f3bc9a4c1c7c41e37927ba47faf639312fc936df3afad7926f5a httpd:2.4.39-alpine],SizeBytes:126894770,},ContainerImage{Names:[httpd@sha256:eb8ccf084cf3e80eece1add239effefd171eb39adbc154d33c14260d905d4060 httpd:2.4.38-alpine],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1],SizeBytes:121711221,},ContainerImage{Names:[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0],SizeBytes:71821984,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/sample-apiserver@sha256:1bafcc6fb1aa990b487850adba9cadc020e42d7905aa8a30481182a477ba24b0 gcr.io/kubernetes-e2e-test-images/sample-apiserver:1.10],SizeBytes:61365829,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727 gcr.io/kubernetes-e2e-test-images/agnhost:2.6],SizeBytes:57345321,},ContainerImage{Names:[quay.io/k8scsi/csi-provisioner@sha256:0efcb424f1dde9b9fb11a1a14f2e48ab47e1c3f08bc3a929990dcfcb1f7ab34f quay.io/k8scsi/csi-provisioner:v1.4.0-rc1],SizeBytes:54431016,},ContainerImage{Names:[quay.io/k8scsi/csi-snapshotter@sha256:e3d3e742e32d00488fdb401045b9b1d033d7ca0ab6e760f77b24750fc95e5f70 quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1],SizeBytes:51703561,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0],SizeBytes:46226754,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:0aba670b4d9d6b2e720bbf575d733156c676b693ca26501235444490300db838 quay.io/k8scsi/csi-attacher:v1.1.0],SizeBytes:42839085,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:7d46fb6eb8b890dc546029d1565d502b4a1d974d33625c6ee2bc7991b77fc1a1 quay.io/k8scsi/csi-resizer:v0.2.0],SizeBytes:42817100,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:f315c9042e56def3c05c6b04fe79ec9da6d39ddc557ca365a76cf35964ea08b6 quay.io/k8scsi/csi-resizer:v0.1.0],SizeBytes:42623056,},ContainerImage{Names:[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1],SizeBytes:42541759,},ContainerImage{Names:[gcr.io/google-containers/debian-base@sha256:6966a0aedd7592c18ff2dd803c08bd85780ee19f5e3a2e7cf908a4cd837afcde gcr.io/google-containers/debian-base:0.4.1],SizeBytes:42323657,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nonroot@sha256:d4ede5c74517090b6686219059118ed178cf4620f5db8781b32f806bb1e7395b gcr.io/kubernetes-e2e-test-images/nonroot:1.0],SizeBytes:42321438,},ContainerImage{Names:[redis@sha256:50899ea1ceed33fa03232f3ac57578a424faa1742c1ac9c7a7bdb95cdf19b858 redis:5.0.5-alpine],SizeBytes:29331594,},ContainerImage{Names:[quay.io/k8scsi/hostpathplugin@sha256:b4826e492fc1762fceaf9726f41575ca0a4567864d3d235da874818de18039de quay.io/k8scsi/hostpathplugin:v1.2.0-rc5],SizeBytes:28761497,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7],SizeBytes:22909487,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/echoserver@sha256:e9ba514b896cdf559eef8788b66c2c3ee55f3572df617647b4b0d8b6bf81cf19 gcr.io/kubernetes-e2e-test-images/echoserver:2.2],SizeBytes:21692741,},ContainerImage{Names:[quay.io/k8scsi/mock-driver@sha256:e0eed916b7d970bad2b7d9875f9ad16932f987f0f3d91ec5d86da68b0b5cc9d1 quay.io/k8scsi/mock-driver:v2.1.0],SizeBytes:16226335,},ContainerImage{Names:[nginx@sha256:485b610fefec7ff6c463ced9623314a04ed67e3945b9c08d7e53a47f6d108dc7 nginx:1.14-alpine],SizeBytes:16032814,},ContainerImage{Names:[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0],SizeBytes:15815995,},ContainerImage{Names:[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0],SizeBytes:14967303,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/dnsutils@sha256:2abeee84efb79c14d731966e034af33bf324d3b26ca28497555511ff094b3ddd gcr.io/kubernetes-e2e-test-images/dnsutils:1.1],SizeBytes:9349974,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nonewprivs@sha256:10066e9039219449fe3c81f38fe01928f87914150768ab81b62a468e51fa7411 gcr.io/kubernetes-e2e-test-images/nonewprivs:1.0],SizeBytes:6757579,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nautilus@sha256:33a732d4c42a266912a5091598a0f07653c9134db4b8d571690d8afd509e0bfc gcr.io/kubernetes-e2e-test-images/nautilus:1.0],SizeBytes:4753501,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/kitten@sha256:bcbc4875c982ab39aa7c4f6acf4a287f604e996d9f34a3fbda8c3d1a7457d1f6 gcr.io/kubernetes-e2e-test-images/kitten:1.0],SizeBytes:4747037,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/test-webserver@sha256:7f93d6e32798ff28bc6289254d0c2867fe2c849c8e46edc50f8624734309812e gcr.io/kubernetes-e2e-test-images/test-webserver:1.0],SizeBytes:4732240,},ContainerImage{Names:[alpine@sha256:8421d9a84432575381bfabd248f1eb56f3aa21d9d7cd2511583c68c9b7511d10 alpine:3.7],SizeBytes:4206494,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 gcr.io/kubernetes-e2e-test-images/mounttest:1.0],SizeBytes:1563521,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest-user@sha256:17319ca525ee003681fccf7e8c6b1b910ff4f49b653d939ac7f9b6e7c463933d gcr.io/kubernetes-e2e-test-images/mounttest-user:1.0],SizeBytes:1450451,},ContainerImage{Names:[busybox@sha256:1303dbf110c57f3edf68d9f5a16c082ec06c4cf7604831669faf2c712260b5a0 busybox:latest],SizeBytes:1219790,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29],SizeBytes:1154361,},ContainerImage{Names:[busybox@sha256:bbc3a03235220b170ba48a157dd097dd1379299370e1ed99ce976df0355d24f0 busybox:1.27],SizeBytes:1129289,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea k8s.gcr.io/pause-amd64:3.1 k8s.gcr.io/pause:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 21:28:01.685: INFO: Logging kubelet events for node k8s-agentpool-24944505-vmss000000 Nov 13 21:28:01.740: INFO: Logging pods the kubelet thinks is on node k8s-agentpool-24944505-vmss000000 Nov 13 21:28:01.826: INFO: external-provisioner-fs2p2 started at 2019-11-13 21:19:00 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:01.826: INFO: Container nfs-provisioner ready: true, restart count 0 Nov 13 21:28:01.826: INFO: keyvault-flexvolume-sn47m started at 2019-11-13 20:42:59 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:01.826: INFO: Container keyvault-flexvolume ready: true, restart count 0 Nov 13 21:28:01.826: INFO: blobfuse-flexvol-installer-drhkj started at 2019-11-13 20:42:59 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:01.826: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 Nov 13 21:28:01.826: INFO: kube-proxy-h9n4c started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:01.826: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 21:28:01.826: INFO: kubernetes-dashboard-65966766b9-lq4lm started at 2019-11-13 20:42:59 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:01.826: INFO: Container kubernetes-dashboard ready: true, restart count 0 Nov 13 21:28:01.826: INFO: metrics-server-58ff8c5ddf-r2d7f started at 2019-11-13 20:42:59 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:01.826: INFO: Container metrics-server ready: true, restart count 0 Nov 13 21:28:01.826: INFO: azure-ip-masq-agent-qksp6 started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:01.826: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 21:28:01.826: INFO: tiller-deploy-7559b6b885-qxwkm started at 2019-11-13 20:42:59 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:01.826: INFO: Container tiller ready: true, restart count 0 W1113 21:28:01.879115 92634 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 21:28:02.018: INFO: Latency metrics for node k8s-agentpool-24944505-vmss000000 Nov 13 21:28:02.018: INFO: Logging node info for node k8s-agentpool-24944505-vmss000001 Nov 13 21:28:02.071: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool-24944505-vmss000001 /api/v1/nodes/k8s-agentpool-24944505-vmss000001 da675d2e-0aad-4a8f-b94d-1311fb000563 59223 0 2019-11-13 20:42:24 +0000 UTC <nil> <nil> map[agentpool:agentpool beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool-24944505-vmss000001 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: storageprofile:managed storagetier:Premium_LRS] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-351":"k8s-agentpool-24944505-vmss000001","csi-hostpath-ephemeral-4187":"k8s-agentpool-24944505-vmss000001","csi-hostpath-ephemeral-5584":"k8s-agentpool-24944505-vmss000001","csi-hostpath-provisioning-1638":"k8s-agentpool-24944505-vmss000001","csi-hostpath-provisioning-225":"k8s-agentpool-24944505-vmss000001","csi-hostpath-provisioning-6430":"k8s-agentpool-24944505-vmss000001","csi-hostpath-provisioning-7099":"k8s-agentpool-24944505-vmss000001","csi-hostpath-volume-6291":"k8s-agentpool-24944505-vmss000001","csi-hostpath-volume-expand-7524":"k8s-agentpool-24944505-vmss000001","csi-hostpath-volume-expand-8458":"k8s-agentpool-24944505-vmss000001","csi-mock-csi-mock-volumes-2938":"csi-mock-csi-mock-volumes-2938","csi-mock-csi-mock-volumes-4469":"csi-mock-csi-mock-volumes-4469","csi-mock-csi-mock-volumes-4959":"csi-mock-csi-mock-volumes-4959","csi-mock-csi-mock-volumes-597":"csi-mock-csi-mock-volumes-597","csi-mock-csi-mock-volumes-8812":"csi-mock-csi-mock-volumes-8812","csi-mock-csi-mock-volumes-9088":"csi-mock-csi-mock-volumes-9088","csi-mock-csi-mock-volumes-9179":"csi-mock-csi-mock-volumes-9179"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-24944505-vmss/virtualMachines/1,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16797569024 0} {<nil>} 16403876Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16011137024 0} {<nil>} 15635876Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 21:27:05 +0000 UTC,LastTransitionTime:2019-11-13 20:42:24 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 21:27:05 +0000 UTC,LastTransitionTime:2019-11-13 20:42:24 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 21:27:05 +0000 UTC,LastTransitionTime:2019-11-13 20:42:24 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 21:27:05 +0000 UTC,LastTransitionTime:2019-11-13 21:27:05 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.248.0.5,},NodeAddress{Type:Hostname,Address:k8s-agentpool-24944505-vmss000001,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:6ed41b2c4b884fccaf94748594531e06,SystemUUID:9CE5C2E3-7C34-C64A-9734-2477E19937B7,BootID:306ac3f6-a23b-423e-83c9-e3d5f3938f6c,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[quay.io/kubernetes_incubator/nfs-provisioner@sha256:df762117e3c891f2d2ddff46ecb0776ba1f9f3c44cfd7739b0683bcd7a7954a8 quay.io/kubernetes_incubator/nfs-provisioner:v2.2.2],SizeBytes:391772778,},ContainerImage{Names:[gluster/glusterdynamic-provisioner@sha256:90067cb05a7d217651e84576935934fd9dcff8e6bcdcbaa416bbf36fcd09dbd1 gluster/glusterdynamic-provisioner:v1.0],SizeBytes:373281573,},ContainerImage{Names:[gcr.io/google-samples/gb-frontend@sha256:35cb427341429fac3df10ff74600ea73e8ec0754d78f9ce89e0b4f3d70d53ba6 gcr.io/google-samples/gb-frontend:v6],SizeBytes:373099368,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/volume/nfs@sha256:c2ad734346f608a5f7d69cfded93c4e8094069320657bd372d12ba21dea3ea71 gcr.io/kubernetes-e2e-test-images/volume/nfs:1.0],SizeBytes:225358913,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/jessie-dnsutils@sha256:ad583e33cb284f7ef046673809b146ec4053cda19b54a85d2b180a86169715eb gcr.io/kubernetes-e2e-test-images/jessie-dnsutils:1.0],SizeBytes:195659796,},ContainerImage{Names:[httpd@sha256:addd70e4ee83f3bc9a4c1c7c41e37927ba47faf639312fc936df3afad7926f5a httpd:2.4.39-alpine],SizeBytes:126894770,},ContainerImage{Names:[httpd@sha256:eb8ccf084cf3e80eece1add239effefd171eb39adbc154d33c14260d905d4060 httpd:2.4.38-alpine],SizeBytes:123781643,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727 gcr.io/kubernetes-e2e-test-images/agnhost:2.6],SizeBytes:57345321,},ContainerImage{Names:[quay.io/k8scsi/csi-provisioner@sha256:0efcb424f1dde9b9fb11a1a14f2e48ab47e1c3f08bc3a929990dcfcb1f7ab34f quay.io/k8scsi/csi-provisioner:v1.4.0-rc1],SizeBytes:54431016,},ContainerImage{Names:[quay.io/k8scsi/csi-snapshotter@sha256:e3d3e742e32d00488fdb401045b9b1d033d7ca0ab6e760f77b24750fc95e5f70 quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1],SizeBytes:51703561,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0],SizeBytes:46226754,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:0aba670b4d9d6b2e720bbf575d733156c676b693ca26501235444490300db838 quay.io/k8scsi/csi-attacher:v1.1.0],SizeBytes:42839085,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:7d46fb6eb8b890dc546029d1565d502b4a1d974d33625c6ee2bc7991b77fc1a1 quay.io/k8scsi/csi-resizer:v0.2.0],SizeBytes:42817100,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:f315c9042e56def3c05c6b04fe79ec9da6d39ddc557ca365a76cf35964ea08b6 quay.io/k8scsi/csi-resizer:v0.1.0],SizeBytes:42623056,},ContainerImage{Names:[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0],SizeBytes:42488424,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nonroot@sha256:d4ede5c74517090b6686219059118ed178cf4620f5db8781b32f806bb1e7395b gcr.io/kubernetes-e2e-test-images/nonroot:1.0],SizeBytes:42321438,},ContainerImage{Names:[redis@sha256:50899ea1ceed33fa03232f3ac57578a424faa1742c1ac9c7a7bdb95cdf19b858 redis:5.0.5-alpine],SizeBytes:29331594,},ContainerImage{Names:[quay.io/k8scsi/hostpathplugin@sha256:b4826e492fc1762fceaf9726f41575ca0a4567864d3d235da874818de18039de quay.io/k8scsi/hostpathplugin:v1.2.0-rc5],SizeBytes:28761497,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7],SizeBytes:22909487,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/echoserver@sha256:e9ba514b896cdf559eef8788b66c2c3ee55f3572df617647b4b0d8b6bf81cf19 gcr.io/kubernetes-e2e-test-images/echoserver:2.2],SizeBytes:21692741,},ContainerImage{Names:[quay.io/k8scsi/mock-driver@sha256:e0eed916b7d970bad2b7d9875f9ad16932f987f0f3d91ec5d86da68b0b5cc9d1 quay.io/k8scsi/mock-driver:v2.1.0],SizeBytes:16226335,},ContainerImage{Names:[nginx@sha256:485b610fefec7ff6c463ced9623314a04ed67e3945b9c08d7e53a47f6d108dc7 nginx:1.14-alpine],SizeBytes:16032814,},ContainerImage{Names:[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0],SizeBytes:15815995,},ContainerImage{Names:[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0],SizeBytes:14967303,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/dnsutils@sha256:2abeee84efb79c14d731966e034af33bf324d3b26ca28497555511ff094b3ddd gcr.io/kubernetes-e2e-test-images/dnsutils:1.1],SizeBytes:9349974,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nautilus@sha256:33a732d4c42a266912a5091598a0f07653c9134db4b8d571690d8afd509e0bfc gcr.io/kubernetes-e2e-test-images/nautilus:1.0],SizeBytes:4753501,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/kitten@sha256:bcbc4875c982ab39aa7c4f6acf4a287f604e996d9f34a3fbda8c3d1a7457d1f6 gcr.io/kubernetes-e2e-test-images/kitten:1.0],SizeBytes:4747037,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/test-webserver@sha256:7f93d6e32798ff28bc6289254d0c2867fe2c849c8e46edc50f8624734309812e gcr.io/kubernetes-e2e-test-images/test-webserver:1.0],SizeBytes:4732240,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:4206620,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 gcr.io/kubernetes-e2e-test-images/mounttest:1.0],SizeBytes:1563521,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest-user@sha256:17319ca525ee003681fccf7e8c6b1b910ff4f49b653d939ac7f9b6e7c463933d gcr.io/kubernetes-e2e-test-images/mounttest-user:1.0],SizeBytes:1450451,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29],SizeBytes:1154361,},ContainerImage{Names:[busybox@sha256:bbc3a03235220b170ba48a157dd097dd1379299370e1ed99ce976df0355d24f0 busybox:1.27],SizeBytes:1129289,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea k8s.gcr.io/pause-amd64:3.1 k8s.gcr.io/pause:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 21:28:02.072: INFO: Logging kubelet events for node k8s-agentpool-24944505-vmss000001 Nov 13 21:28:02.127: INFO: Logging pods the kubelet thinks is on node k8s-agentpool-24944505-vmss000001 Nov 13 21:28:02.254: INFO: azure-ip-masq-agent-wwpg7 started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:02.254: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 21:28:02.254: INFO: csi-hostpathplugin-0 started at 2019-11-13 21:19:07 +0000 UTC (0+3 container statuses recorded) Nov 13 21:28:02.254: INFO: Container hostpath ready: true, restart count 0 Nov 13 21:28:02.254: INFO: Container liveness-probe ready: true, restart count 0 Nov 13 21:28:02.254: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 13 21:28:02.254: INFO: kube-proxy-7bpf7 started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:02.254: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 21:28:02.254: INFO: hostexec-k8s-agentpool-24944505-vmss000001 started at 2019-11-13 21:19:02 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:02.254: INFO: Container agnhost ready: true, restart count 0 Nov 13 21:28:02.254: INFO: external-provisioner-rqq6k started at 2019-11-13 21:19:21 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:02.254: INFO: Container nfs-provisioner ready: true, restart count 0 Nov 13 21:28:02.254: INFO: recycler-for-nfs-kbrgv started at 2019-11-13 21:19:57 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:02.254: INFO: Container pv-recycler ready: false, restart count 0 Nov 13 21:28:02.254: INFO: csi-hostpath-resizer-0 started at 2019-11-13 21:19:09 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:02.255: INFO: Container csi-resizer ready: true, restart count 0 Nov 13 21:28:02.255: INFO: affinity-nodeport-transition-24rzm started at 2019-11-13 21:19:04 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:02.255: INFO: Container affinity-nodeport-transition ready: true, restart count 0 Nov 13 21:28:02.255: INFO: csi-hostpath-attacher-0 started at 2019-11-13 21:19:05 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:02.255: INFO: Container csi-attacher ready: true, restart count 0 Nov 13 21:28:02.255: INFO: pod-subpath-test-csi-hostpath-dynamicpv-2c99 started at 2019-11-13 21:20:22 +0000 UTC (2+1 container statuses recorded) Nov 13 21:28:02.255: INFO: Init container init-volume-csi-hostpath-dynamicpv-2c99 ready: true, restart count 0 Nov 13 21:28:02.255: INFO: Init container test-init-subpath-csi-hostpath-dynamicpv-2c99 ready: false, restart count 0 Nov 13 21:28:02.255: INFO: Container test-container-subpath-csi-hostpath-dynamicpv-2c99 ready: false, restart count 0 Nov 13 21:28:02.255: INFO: pod-subpath-test-nfs-dynamicpv-zcz2 started at 2019-11-13 21:20:28 +0000 UTC (1+1 container statuses recorded) Nov 13 21:28:02.255: INFO: Init container init-volume-nfs-dynamicpv-zcz2 ready: true, restart count 0 Nov 13 21:28:02.255: INFO: Container test-container-subpath-nfs-dynamicpv-zcz2 ready: false, restart count 0 Nov 13 21:28:02.255: INFO: pod-projected-secrets-2fc08520-7bb7-4fea-8bde-adf3354ecd89 started at 2019-11-13 21:19:00 +0000 UTC (0+3 container statuses recorded) Nov 13 21:28:02.255: INFO: Container creates-volume-test ready: true, restart count 0 Nov 13 21:28:02.255: INFO: Container dels-volume-test ready: true, restart count 0 Nov 13 21:28:02.255: INFO: Container upds-volume-test ready: true, restart count 0 Nov 13 21:28:02.255: INFO: privileged-pod started at 2019-11-13 21:20:06 +0000 UTC (0+2 container statuses recorded) Nov 13 21:28:02.255: INFO: Container not-privileged-container ready: true, restart count 0 Nov 13 21:28:02.255: INFO: Container privileged-container ready: true, restart count 0 Nov 13 21:28:02.255: INFO: pfpod started at 2019-11-13 21:20:18 +0000 UTC (0+2 container statuses recorded) Nov 13 21:28:02.255: INFO: Container portforwardtester ready: true, restart count 0 Nov 13 21:28:02.255: INFO: Container readiness ready: false, restart count 0 Nov 13 21:28:02.255: INFO: hostexec-k8s-agentpool-24944505-vmss000001 started at 2019-11-13 21:17:56 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:02.255: INFO: Container agnhost ready: true, restart count 0 Nov 13 21:28:02.255: INFO: csi-snapshotter-0 started at 2019-11-13 21:19:10 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:02.255: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 13 21:28:02.255: INFO: pod-subpath-test-local-preprovisionedpv-lglz started at 2019-11-13 21:20:26 +0000 UTC (2+2 container statuses recorded) Nov 13 21:28:02.255: INFO: Init container init-volume-local-preprovisionedpv-lglz ready: true, restart count 0 Nov 13 21:28:02.255: INFO: Init container test-init-subpath-local-preprovisionedpv-lglz ready: false, restart count 0 Nov 13 21:28:02.255: INFO: Container test-container-subpath-local-preprovisionedpv-lglz ready: false, restart count 0 Nov 13 21:28:02.255: INFO: Container test-container-volume-local-preprovisionedpv-lglz ready: false, restart count 0 Nov 13 21:28:02.255: INFO: coredns-87f5d796-jd8bc started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:02.255: INFO: Container coredns ready: true, restart count 0 Nov 13 21:28:02.255: INFO: keyvault-flexvolume-7phpr started at 2019-11-13 20:42:58 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:02.255: INFO: Container keyvault-flexvolume ready: true, restart count 0 Nov 13 21:28:02.255: INFO: pod-subpath-test-local-preprovisionedpv-rw7l started at 2019-11-13 21:18:51 +0000 UTC (2+1 container statuses recorded) Nov 13 21:28:02.255: INFO: Init container init-volume-local-preprovisionedpv-rw7l ready: true, restart count 0 Nov 13 21:28:02.255: INFO: Init container test-init-subpath-local-preprovisionedpv-rw7l ready: true, restart count 0 Nov 13 21:28:02.255: INFO: Container test-container-subpath-local-preprovisionedpv-rw7l ready: false, restart count 0 Nov 13 21:28:02.255: INFO: busybox-host-aliases7e358483-64e0-4e66-8435-049f1149ff6a started at 2019-11-13 21:19:26 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:02.255: INFO: Container busybox-host-aliases7e358483-64e0-4e66-8435-049f1149ff6a ready: true, restart count 0 Nov 13 21:28:02.255: INFO: csi-hostpath-provisioner-0 started at 2019-11-13 21:19:08 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:02.255: INFO: Container csi-provisioner ready: true, restart count 0 Nov 13 21:28:02.255: INFO: blobfuse-flexvol-installer-cpm6h started at 2019-11-13 20:42:58 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:02.255: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 Nov 13 21:28:02.255: INFO: external-provisioner-qms6n started at 2019-11-13 21:19:56 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:02.255: INFO: Container nfs-provisioner ready: true, restart count 0 W1113 21:28:02.307231 92634 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 21:28:02.462: INFO: Latency metrics for node k8s-agentpool-24944505-vmss000001 Nov 13 21:28:02.462: INFO: Logging node info for node k8s-master-24944505-vmss000000 Nov 13 21:28:02.515: INFO: Node Info: &Node{ObjectMeta:{k8s-master-24944505-vmss000000 /api/v1/nodes/k8s-master-24944505-vmss000000 048fbae2-b97f-43a2-b471-398180136f12 59252 0 2019-11-13 20:42:13 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-24944505-vmss000000 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-24944505-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284887552 0} {<nil>} 7114148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6498455552 0} {<nil>} 6346148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 21:27:10 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 21:27:10 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 21:27:10 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 21:27:10 +0000 UTC,LastTransitionTime:2019-11-13 20:42:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.4,},NodeAddress{Type:Hostname,Address:k8s-master-24944505-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:7bd754c0dc8647518c3043bcae3a76cc,SystemUUID:FA5CFE99-8148-6C46-B4E6-5EA2A5DE9D2A,BootID:74f7b0a2-e85c-438e-a642-c5b32900bb28,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:3237a53b135d8692ade82eca845fd0259ecbfcbe05fad35400e7d5c2018c4abd k8sprow.azurecr.io/azure-cloud-controller-manager:1194710255657816064-e688ecb0],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 21:28:02.515: INFO: Logging kubelet events for node k8s-master-24944505-vmss000000 Nov 13 21:28:02.570: INFO: Logging pods the kubelet thinks is on node k8s-master-24944505-vmss000000 Nov 13 21:28:02.646: INFO: kube-proxy-d9jjh started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:02.646: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 21:28:02.646: INFO: cloud-controller-manager-k8s-master-24944505-vmss000000 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:02.646: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 13 21:28:02.646: INFO: kube-addon-manager-k8s-master-24944505-vmss000000 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:02.646: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 13 21:28:02.646: INFO: kube-apiserver-k8s-master-24944505-vmss000000 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:02.646: INFO: Container kube-apiserver ready: true, restart count 0 Nov 13 21:28:02.646: INFO: kube-controller-manager-k8s-master-24944505-vmss000000 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:02.646: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 13 21:28:02.646: INFO: kube-scheduler-k8s-master-24944505-vmss000000 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:02.646: INFO: Container kube-scheduler ready: true, restart count 0 Nov 13 21:28:02.646: INFO: azure-ip-masq-agent-6pnst started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:02.646: INFO: Container azure-ip-masq-agent ready: true, restart count 0 W1113 21:28:02.699070 92634 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 21:28:02.820: INFO: Latency metrics for node k8s-master-24944505-vmss000000 Nov 13 21:28:02.820: INFO: Logging node info for node k8s-master-24944505-vmss000001 Nov 13 21:28:02.871: INFO: Node Info: &Node{ObjectMeta:{k8s-master-24944505-vmss000001 /api/v1/nodes/k8s-master-24944505-vmss000001 a2619878-b119-44f9-b89f-4073bebbb56c 59217 0 2019-11-13 20:42:40 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-24944505-vmss000001 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.6.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-24944505-vmss/virtualMachines/1,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.6.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284883456 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6498451456 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:43:08 +0000 UTC,LastTransitionTime:2019-11-13 20:43:08 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 21:27:06 +0000 UTC,LastTransitionTime:2019-11-13 20:42:34 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 21:27:06 +0000 UTC,LastTransitionTime:2019-11-13 20:42:34 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 21:27:06 +0000 UTC,LastTransitionTime:2019-11-13 20:42:34 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 21:27:06 +0000 UTC,LastTransitionTime:2019-11-13 20:42:40 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.5,},NodeAddress{Type:Hostname,Address:k8s-master-24944505-vmss000001,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:c866a7c1ed64481990416e24387dd317,SystemUUID:EBE64557-2AC8-5740-9FFE-AB9F12412FBC,BootID:3dbb0e0e-0479-4821-bc3f-a8269d250e4c,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:3237a53b135d8692ade82eca845fd0259ecbfcbe05fad35400e7d5c2018c4abd k8sprow.azurecr.io/azure-cloud-controller-manager:1194710255657816064-e688ecb0],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 21:28:02.872: INFO: Logging kubelet events for node k8s-master-24944505-vmss000001 Nov 13 21:28:02.928: INFO: Logging pods the kubelet thinks is on node k8s-master-24944505-vmss000001 Nov 13 21:28:03.004: INFO: cloud-controller-manager-k8s-master-24944505-vmss000001 started at 2019-11-13 20:42:28 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:03.004: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 13 21:28:03.004: INFO: kube-addon-manager-k8s-master-24944505-vmss000001 started at 2019-11-13 20:42:28 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:03.004: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 13 21:28:03.004: INFO: kube-apiserver-k8s-master-24944505-vmss000001 started at 2019-11-13 20:42:28 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:03.004: INFO: Container kube-apiserver ready: true, restart count 0 Nov 13 21:28:03.004: INFO: kube-controller-manager-k8s-master-24944505-vmss000001 started at 2019-11-13 20:42:28 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:03.004: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 13 21:28:03.004: INFO: kube-proxy-w4bz5 started at 2019-11-13 20:42:41 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:03.004: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 21:28:03.004: INFO: azure-ip-masq-agent-bs52d started at 2019-11-13 20:42:41 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:03.004: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 21:28:03.004: INFO: kube-scheduler-k8s-master-24944505-vmss000001 started at 2019-11-13 20:42:28 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:03.004: INFO: Container kube-scheduler ready: true, restart count 0 W1113 21:28:03.056652 92634 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 21:28:03.177: INFO: Latency metrics for node k8s-master-24944505-vmss000001 Nov 13 21:28:03.177: INFO: Logging node info for node k8s-master-24944505-vmss000002 Nov 13 21:28:03.228: INFO: Node Info: &Node{ObjectMeta:{k8s-master-24944505-vmss000002 /api/v1/nodes/k8s-master-24944505-vmss000002 f3b05267-a009-4dba-9f97-4f772e48707e 59250 0 2019-11-13 20:42:13 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-24944505-vmss000002 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.4.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-24944505-vmss/virtualMachines/2,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284887552 0} {<nil>} 7114148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6498455552 0} {<nil>} 6346148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 21:27:09 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 21:27:09 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 21:27:09 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 21:27:09 +0000 UTC,LastTransitionTime:2019-11-13 20:42:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.6,},NodeAddress{Type:Hostname,Address:k8s-master-24944505-vmss000002,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:fc1777f37ced4b9c97acb9b30e276c4f,SystemUUID:E9DF19DC-D02C-714C-A7D6-0BA6CE959C55,BootID:622958a8-d45d-4edf-9266-4dbb759e2359,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:3237a53b135d8692ade82eca845fd0259ecbfcbe05fad35400e7d5c2018c4abd k8sprow.azurecr.io/azure-cloud-controller-manager:1194710255657816064-e688ecb0],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 21:28:03.228: INFO: Logging kubelet events for node k8s-master-24944505-vmss000002 Nov 13 21:28:03.283: INFO: Logging pods the kubelet thinks is on node k8s-master-24944505-vmss000002 Nov 13 21:28:03.355: INFO: kube-apiserver-k8s-master-24944505-vmss000002 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:03.355: INFO: Container kube-apiserver ready: true, restart count 0 Nov 13 21:28:03.355: INFO: kube-controller-manager-k8s-master-24944505-vmss000002 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:03.355: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 13 21:28:03.355: INFO: kube-scheduler-k8s-master-24944505-vmss000002 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:03.355: INFO: Container kube-scheduler ready: true, restart count 0 Nov 13 21:28:03.355: INFO: cloud-controller-manager-k8s-master-24944505-vmss000002 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:03.355: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 13 21:28:03.355: INFO: azure-ip-masq-agent-6skf2 started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:03.355: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 21:28:03.355: INFO: kube-proxy-xwz78 started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:03.355: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 21:28:03.355: INFO: kube-addon-manager-k8s-master-24944505-vmss000002 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:03.355: INFO: Container kube-addon-manager ready: true, restart count 0 W1113 21:28:03.408043 92634 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 21:28:03.527: INFO: Latency metrics for node k8s-master-24944505-vmss000002 Nov 13 21:28:03.527: INFO: Logging node info for node k8s-master-24944505-vmss000003 Nov 13 21:28:03.579: INFO: Node Info: &Node{ObjectMeta:{k8s-master-24944505-vmss000003 /api/v1/nodes/k8s-master-24944505-vmss000003 888cc19e-c62d-4945-a425-97d14e38a0b8 59471 0 2019-11-13 20:42:35 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-24944505-vmss000003 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.5.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-24944505-vmss/virtualMachines/3,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.5.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284887552 0} {<nil>} 7114148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6498455552 0} {<nil>} 6346148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 21:28:02 +0000 UTC,LastTransitionTime:2019-11-13 20:42:29 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 21:28:02 +0000 UTC,LastTransitionTime:2019-11-13 20:42:29 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 21:28:02 +0000 UTC,LastTransitionTime:2019-11-13 20:42:29 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 21:28:02 +0000 UTC,LastTransitionTime:2019-11-13 20:42:35 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.7,},NodeAddress{Type:Hostname,Address:k8s-master-24944505-vmss000003,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:2da5fe5a442b4e178a2b62ba91366e4b,SystemUUID:D2D1E816-D70E-3D40-B959-B00A2B76A83A,BootID:c5929aa1-3dac-45d5-bb2c-174c73c7300e,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:3237a53b135d8692ade82eca845fd0259ecbfcbe05fad35400e7d5c2018c4abd k8sprow.azurecr.io/azure-cloud-controller-manager:1194710255657816064-e688ecb0],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 21:28:03.579: INFO: Logging kubelet events for node k8s-master-24944505-vmss000003 Nov 13 21:28:03.634: INFO: Logging pods the kubelet thinks is on node k8s-master-24944505-vmss000003 Nov 13 21:28:03.708: INFO: azure-ip-masq-agent-nlqss started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:03.708: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 21:28:03.708: INFO: kube-proxy-kgh5h started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:03.708: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 21:28:03.708: INFO: kube-addon-manager-k8s-master-24944505-vmss000003 started at 2019-11-13 20:42:22 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:03.708: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 13 21:28:03.708: INFO: kube-apiserver-k8s-master-24944505-vmss000003 started at 2019-11-13 20:42:23 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:03.708: INFO: Container kube-apiserver ready: true, restart count 0 Nov 13 21:28:03.708: INFO: kube-controller-manager-k8s-master-24944505-vmss000003 started at 2019-11-13 20:42:23 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:03.708: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 13 21:28:03.708: INFO: kube-scheduler-k8s-master-24944505-vmss000003 started at 2019-11-13 20:42:23 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:03.708: INFO: Container kube-scheduler ready: true, restart count 0 Nov 13 21:28:03.708: INFO: cloud-controller-manager-k8s-master-24944505-vmss000003 started at 2019-11-13 20:42:44 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:03.708: INFO: Container cloud-controller-manager ready: true, restart count 0 W1113 21:28:03.761725 92634 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 21:28:03.892: INFO: Latency metrics for node k8s-master-24944505-vmss000003 Nov 13 21:28:03.892: INFO: Logging node info for node k8s-master-24944505-vmss000004 Nov 13 21:28:03.944: INFO: Node Info: &Node{ObjectMeta:{k8s-master-24944505-vmss000004 /api/v1/nodes/k8s-master-24944505-vmss000004 a26d844f-0612-406b-9a65-7e2ac952a2d5 59251 0 2019-11-13 20:42:14 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-24944505-vmss000004 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-24944505-vmss/virtualMachines/4,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284887552 0} {<nil>} 7114148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6498455552 0} {<nil>} 6346148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 21:27:10 +0000 UTC,LastTransitionTime:2019-11-13 20:42:09 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 21:27:10 +0000 UTC,LastTransitionTime:2019-11-13 20:42:09 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 21:27:10 +0000 UTC,LastTransitionTime:2019-11-13 20:42:09 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 21:27:10 +0000 UTC,LastTransitionTime:2019-11-13 20:42:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.8,},NodeAddress{Type:Hostname,Address:k8s-master-24944505-vmss000004,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:2208959aca1045d9b48fce8d784c1e41,SystemUUID:E253DB3A-3A4D-5648-8169-B2E2976CFB4B,BootID:09bc6101-2da9-4c8e-8274-43db8c5fe1cf,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:3237a53b135d8692ade82eca845fd0259ecbfcbe05fad35400e7d5c2018c4abd k8sprow.azurecr.io/azure-cloud-controller-manager:1194710255657816064-e688ecb0],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 21:28:03.944: INFO: Logging kubelet events for node k8s-master-24944505-vmss000004 Nov 13 21:28:04.000: INFO: Logging pods the kubelet thinks is on node k8s-master-24944505-vmss000004 Nov 13 21:28:04.073: INFO: cloud-controller-manager-k8s-master-24944505-vmss000004 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:04.073: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 13 21:28:04.073: INFO: kube-addon-manager-k8s-master-24944505-vmss000004 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:04.073: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 13 21:28:04.073: INFO: kube-apiserver-k8s-master-24944505-vmss000004 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:04.073: INFO: Container kube-apiserver ready: true, restart count 0 Nov 13 21:28:04.073: INFO: kube-controller-manager-k8s-master-24944505-vmss000004 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:04.073: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 13 21:28:04.073: INFO: kube-scheduler-k8s-master-24944505-vmss000004 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:04.073: INFO: Container kube-scheduler ready: true, restart count 0 Nov 13 21:28:04.073: INFO: azure-ip-masq-agent-xnw2h started at 2019-11-13 20:42:38 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:04.073: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 21:28:04.073: INFO: kube-proxy-fld6p started at 2019-11-13 20:42:38 +0000 UTC (0+1 container statuses recorded) Nov 13 21:28:04.073: INFO: Container kube-proxy ready: true, restart count 0 W1113 21:28:04.126812 92634 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 21:28:04.248: INFO: Latency metrics for node k8s-master-24944505-vmss000004 Nov 13 21:28:04.248: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "port-forwarding-905" for this suite. Nov 13 21:38:04.517: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered Nov 13 21:38:06.172: INFO: namespace: port-forwarding-905, resource: pods, items remaining: 1 Nov 13 21:38:06.277: INFO: namespace: port-forwarding-905, DeletionTimetamp: 2019-11-13 21:28:04 +0000 UTC, Finalizers: [kubernetes], Phase: Terminating Nov 13 21:38:06.328: INFO: namespace: port-forwarding-905, total namespaces: 11, active: 7, terminating: 4 Nov 13 21:38:06.380: INFO: POD NODE PHASE GRACE CONDITIONS Nov 13 21:38:06.380: INFO: pfpod k8s-agentpool-24944505-vmss000001 Running 30s [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:20:18 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:23:02 +0000 UTC ContainersNotReady containers with unready status: [readiness portforwardtester]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:23:02 +0000 UTC ContainersNotReady containers with unready status: [readiness portforwardtester]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:20:09 +0000 UTC }] Nov 13 21:38:06.380: INFO: Nov 13 21:38:06.380: INFO: Couldn't delete ns: "port-forwarding-905": namespace port-forwarding-905 was not deleted with limit: timed out waiting for the condition, pods remaining: 1 (&errors.errorString{s:"namespace port-forwarding-905 was not deleted with limit: timed out waiting for the condition, pods remaining: 1"})
Find pfpod mentions in log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[sig\-scheduling\]\sPreemptionExecutionPath\sruns\sReplicaSets\sto\sverify\spreemption\srunning\spath$'
test/e2e/scheduling/preemption.go:345 Nov 13 20:57:12.636: Unexpected error: <*errors.errorString | 0xc00200c810>: { s: "replicaset \"rs-pod1\" never had desired number of .status.availableReplicas", } replicaset "rs-pod1" never had desired number of .status.availableReplicas occurred test/e2e/scheduling/preemption.go:510from junit_06.xml
[BeforeEach] [sig-scheduling] PreemptionExecutionPath test/e2e/framework/framework.go:151 �[1mSTEP�[0m: Creating a kubernetes client Nov 13 20:55:45.230: INFO: >>> kubeConfig: /workspace/aks880331199/kubeconfig/kubeconfig.westus2.json �[1mSTEP�[0m: Building a namespace api object, basename sched-preemption-path �[1mSTEP�[0m: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in sched-preemption-path-7365 �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace [BeforeEach] [sig-scheduling] PreemptionExecutionPath test/e2e/scheduling/preemption.go:302 �[1mSTEP�[0m: Finding an available node �[1mSTEP�[0m: Trying to launch a pod without a label to get a node which can launch it. �[1mSTEP�[0m: Explicitly delete pod here to free the resource it takes. Nov 13 20:56:12.040: INFO: found a healthy node: k8s-agentpool-24944505-vmss000000 [It] runs ReplicaSets to verify preemption running path test/e2e/scheduling/preemption.go:345 Nov 13 20:57:12.636: FAIL: Unexpected error: <*errors.errorString | 0xc00200c810>: { s: "replicaset \"rs-pod1\" never had desired number of .status.availableReplicas", } replicaset "rs-pod1" never had desired number of .status.availableReplicas occurred [AfterEach] [sig-scheduling] PreemptionExecutionPath test/e2e/framework/framework.go:152 �[1mSTEP�[0m: Collecting events from namespace "sched-preemption-path-7365". �[1mSTEP�[0m: Found 30 events. Nov 13 20:57:12.745: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-4cp5c: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 13 20:57:12.745: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-4cp5c: {default-scheduler } Scheduled: Successfully assigned sched-preemption-path-7365/rs-pod1-4cp5c to k8s-agentpool-24944505-vmss000000 Nov 13 20:57:12.745: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-4cp5c: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 13 20:57:12.745: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-5b5m7: {default-scheduler } Scheduled: Successfully assigned sched-preemption-path-7365/rs-pod1-5b5m7 to k8s-agentpool-24944505-vmss000000 Nov 13 20:57:12.745: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-5b5m7: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 13 20:57:12.745: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-5b5m7: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 13 20:57:12.745: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-885q4: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 13 20:57:12.745: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-885q4: {default-scheduler } Scheduled: Successfully assigned sched-preemption-path-7365/rs-pod1-885q4 to k8s-agentpool-24944505-vmss000000 Nov 13 20:57:12.745: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-885q4: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 13 20:57:12.745: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-ppppq: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 13 20:57:12.745: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-ppppq: {default-scheduler } Scheduled: Successfully assigned sched-preemption-path-7365/rs-pod1-ppppq to k8s-agentpool-24944505-vmss000000 Nov 13 20:57:12.745: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-ppppq: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 13 20:57:12.745: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-qt8hr: {default-scheduler } Scheduled: Successfully assigned sched-preemption-path-7365/rs-pod1-qt8hr to k8s-agentpool-24944505-vmss000000 Nov 13 20:57:12.745: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-qt8hr: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 13 20:57:12.745: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for without-label: {default-scheduler } Scheduled: Successfully assigned sched-preemption-path-7365/without-label to k8s-agentpool-24944505-vmss000000 Nov 13 20:57:12.745: INFO: At 2019-11-13 20:55:50 +0000 UTC - event for without-label: {kubelet k8s-agentpool-24944505-vmss000000} Pulling: Pulling image "k8s.gcr.io/pause:3.1" Nov 13 20:57:12.745: INFO: At 2019-11-13 20:55:51 +0000 UTC - event for without-label: {kubelet k8s-agentpool-24944505-vmss000000} Pulled: Successfully pulled image "k8s.gcr.io/pause:3.1" Nov 13 20:57:12.745: INFO: At 2019-11-13 20:55:52 +0000 UTC - event for without-label: {kubelet k8s-agentpool-24944505-vmss000000} Created: Created container without-label Nov 13 20:57:12.745: INFO: At 2019-11-13 20:55:53 +0000 UTC - event for without-label: {kubelet k8s-agentpool-24944505-vmss000000} Started: Started container without-label Nov 13 20:57:12.745: INFO: At 2019-11-13 20:56:12 +0000 UTC - event for rs-pod1: {replicaset-controller } SuccessfulCreate: Created pod: rs-pod1-qt8hr Nov 13 20:57:12.745: INFO: At 2019-11-13 20:56:12 +0000 UTC - event for rs-pod1: {replicaset-controller } SuccessfulCreate: Created pod: rs-pod1-4cp5c Nov 13 20:57:12.745: INFO: At 2019-11-13 20:56:12 +0000 UTC - event for rs-pod1: {replicaset-controller } SuccessfulCreate: Created pod: rs-pod1-885q4 Nov 13 20:57:12.745: INFO: At 2019-11-13 20:56:12 +0000 UTC - event for rs-pod1: {replicaset-controller } SuccessfulCreate: Created pod: rs-pod1-ppppq Nov 13 20:57:12.745: INFO: At 2019-11-13 20:56:12 +0000 UTC - event for rs-pod1: {replicaset-controller } SuccessfulCreate: Created pod: rs-pod1-5b5m7 Nov 13 20:57:12.745: INFO: At 2019-11-13 20:56:22 +0000 UTC - event for without-label: {kubelet k8s-agentpool-24944505-vmss000000} Killing: Stopping container without-label Nov 13 20:57:12.745: INFO: At 2019-11-13 20:56:30 +0000 UTC - event for rs-pod1-ppppq: {kubelet k8s-agentpool-24944505-vmss000000} Pulling: Pulling image "k8s.gcr.io/pause:3.1" Nov 13 20:57:12.745: INFO: At 2019-11-13 20:56:31 +0000 UTC - event for rs-pod1-885q4: {kubelet k8s-agentpool-24944505-vmss000000} Pulling: Pulling image "k8s.gcr.io/pause:3.1" Nov 13 20:57:12.745: INFO: At 2019-11-13 20:56:34 +0000 UTC - event for rs-pod1-5b5m7: {kubelet k8s-agentpool-24944505-vmss000000} Pulling: Pulling image "k8s.gcr.io/pause:3.1" Nov 13 20:57:12.745: INFO: At 2019-11-13 20:56:36 +0000 UTC - event for rs-pod1-4cp5c: {kubelet k8s-agentpool-24944505-vmss000000} Pulling: Pulling image "k8s.gcr.io/pause:3.1" Nov 13 20:57:12.745: INFO: At 2019-11-13 20:56:36 +0000 UTC - event for rs-pod1-qt8hr: {kubelet k8s-agentpool-24944505-vmss000000} Pulling: Pulling image "k8s.gcr.io/pause:3.1" Nov 13 20:57:12.801: INFO: POD NODE PHASE GRACE CONDITIONS Nov 13 20:57:12.801: INFO: rs-pod1-4cp5c k8s-agentpool-24944505-vmss000000 Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:56:23 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:56:23 +0000 UTC ContainersNotReady containers with unready status: [pod1]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:56:23 +0000 UTC ContainersNotReady containers with unready status: [pod1]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:56:17 +0000 UTC }] Nov 13 20:57:12.801: INFO: rs-pod1-5b5m7 k8s-agentpool-24944505-vmss000000 Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:56:24 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:56:24 +0000 UTC ContainersNotReady containers with unready status: [pod1]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:56:24 +0000 UTC ContainersNotReady containers with unready status: [pod1]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:56:17 +0000 UTC }] Nov 13 20:57:12.801: INFO: rs-pod1-885q4 k8s-agentpool-24944505-vmss000000 Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:56:22 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:56:22 +0000 UTC ContainersNotReady containers with unready status: [pod1]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:56:22 +0000 UTC ContainersNotReady containers with unready status: [pod1]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:56:17 +0000 UTC }] Nov 13 20:57:12.801: INFO: rs-pod1-ppppq k8s-agentpool-24944505-vmss000000 Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:56:23 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:56:23 +0000 UTC ContainersNotReady containers with unready status: [pod1]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:56:23 +0000 UTC ContainersNotReady containers with unready status: [pod1]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:56:17 +0000 UTC }] Nov 13 20:57:12.801: INFO: rs-pod1-qt8hr k8s-agentpool-24944505-vmss000000 Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:56:24 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:56:24 +0000 UTC ContainersNotReady containers with unready status: [pod1]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:56:24 +0000 UTC ContainersNotReady containers with unready status: [pod1]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:56:17 +0000 UTC }] Nov 13 20:57:12.801: INFO: Nov 13 20:57:12.968: INFO: Logging node info for node k8s-agentpool-24944505-vmss000000 Nov 13 20:57:13.023: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool-24944505-vmss000000 /api/v1/nodes/k8s-agentpool-24944505-vmss000000 5bc7cf9e-80da-4e26-a2c2-20757e870d68 26243 0 2019-11-13 20:42:13 +0000 UTC <nil> <nil> map[agentpool:agentpool beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool-24944505-vmss000000 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: storageprofile:managed storagetier:Premium_LRS] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-1137":"k8s-agentpool-24944505-vmss000000","csi-hostpath-ephemeral-9135":"k8s-agentpool-24944505-vmss000000","csi-hostpath-provisioning-716":"k8s-agentpool-24944505-vmss000000","csi-hostpath-volume-expand-4561":"k8s-agentpool-24944505-vmss000000","csi-hostpath-volume-expand-9975":"k8s-agentpool-24944505-vmss000000","csi-mock-csi-mock-volumes-2863":"csi-mock-csi-mock-volumes-2863","csi-mock-csi-mock-volumes-5360":"csi-mock-csi-mock-volumes-5360","csi-mock-csi-mock-volumes-8606":"csi-mock-csi-mock-volumes-8606"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-24944505-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},example.com/fakecpu: {{800 0} {<nil>} 800 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16797569024 0} {<nil>} 16403876Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},example.com/fakecpu: {{800 0} {<nil>} 800 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16011137024 0} {<nil>} 15635876Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 20:56:17 +0000 UTC,LastTransitionTime:2019-11-13 20:42:03 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 20:56:17 +0000 UTC,LastTransitionTime:2019-11-13 20:42:03 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 20:56:17 +0000 UTC,LastTransitionTime:2019-11-13 20:42:03 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 20:56:17 +0000 UTC,LastTransitionTime:2019-11-13 20:42:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.248.0.4,},NodeAddress{Type:Hostname,Address:k8s-agentpool-24944505-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:0a2e8938ebfc440180ed5c0b5a9edbe7,SystemUUID:8075A1F3-D764-7248-9570-A764E51D7FF2,BootID:4f2b3cbe-c61c-4c6c-93ab-18fca0df5bc5,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[quay.io/kubernetes_incubator/nfs-provisioner@sha256:df762117e3c891f2d2ddff46ecb0776ba1f9f3c44cfd7739b0683bcd7a7954a8 quay.io/kubernetes_incubator/nfs-provisioner:v2.2.2],SizeBytes:391772778,},ContainerImage{Names:[k8s.gcr.io/etcd@sha256:12c2c5e5731c3bcd56e6f1c05c0f9198b6f06793fa7fca2fb43aab9622dc4afa k8s.gcr.io/etcd:3.3.15],SizeBytes:246640776,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/volume/nfs@sha256:c2ad734346f608a5f7d69cfded93c4e8094069320657bd372d12ba21dea3ea71 gcr.io/kubernetes-e2e-test-images/volume/nfs:1.0],SizeBytes:225358913,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/jessie-dnsutils@sha256:ad583e33cb284f7ef046673809b146ec4053cda19b54a85d2b180a86169715eb gcr.io/kubernetes-e2e-test-images/jessie-dnsutils:1.0],SizeBytes:195659796,},ContainerImage{Names:[httpd@sha256:addd70e4ee83f3bc9a4c1c7c41e37927ba47faf639312fc936df3afad7926f5a httpd:2.4.39-alpine],SizeBytes:126894770,},ContainerImage{Names:[httpd@sha256:eb8ccf084cf3e80eece1add239effefd171eb39adbc154d33c14260d905d4060 httpd:2.4.38-alpine],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1],SizeBytes:121711221,},ContainerImage{Names:[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0],SizeBytes:71821984,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/sample-apiserver@sha256:1bafcc6fb1aa990b487850adba9cadc020e42d7905aa8a30481182a477ba24b0 gcr.io/kubernetes-e2e-test-images/sample-apiserver:1.10],SizeBytes:61365829,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727 gcr.io/kubernetes-e2e-test-images/agnhost:2.6],SizeBytes:57345321,},ContainerImage{Names:[quay.io/k8scsi/csi-provisioner@sha256:0efcb424f1dde9b9fb11a1a14f2e48ab47e1c3f08bc3a929990dcfcb1f7ab34f quay.io/k8scsi/csi-provisioner:v1.4.0-rc1],SizeBytes:54431016,},ContainerImage{Names:[quay.io/k8scsi/csi-snapshotter@sha256:e3d3e742e32d00488fdb401045b9b1d033d7ca0ab6e760f77b24750fc95e5f70 quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1],SizeBytes:51703561,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0],SizeBytes:46226754,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:0aba670b4d9d6b2e720bbf575d733156c676b693ca26501235444490300db838 quay.io/k8scsi/csi-attacher:v1.1.0],SizeBytes:42839085,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:7d46fb6eb8b890dc546029d1565d502b4a1d974d33625c6ee2bc7991b77fc1a1 quay.io/k8scsi/csi-resizer:v0.2.0],SizeBytes:42817100,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:f315c9042e56def3c05c6b04fe79ec9da6d39ddc557ca365a76cf35964ea08b6 quay.io/k8scsi/csi-resizer:v0.1.0],SizeBytes:42623056,},ContainerImage{Names:[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1],SizeBytes:42541759,},ContainerImage{Names:[gcr.io/google-containers/debian-base@sha256:6966a0aedd7592c18ff2dd803c08bd85780ee19f5e3a2e7cf908a4cd837afcde gcr.io/google-containers/debian-base:0.4.1],SizeBytes:42323657,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nonroot@sha256:d4ede5c74517090b6686219059118ed178cf4620f5db8781b32f806bb1e7395b gcr.io/kubernetes-e2e-test-images/nonroot:1.0],SizeBytes:42321438,},ContainerImage{Names:[redis@sha256:50899ea1ceed33fa03232f3ac57578a424faa1742c1ac9c7a7bdb95cdf19b858 redis:5.0.5-alpine],SizeBytes:29331594,},ContainerImage{Names:[quay.io/k8scsi/hostpathplugin@sha256:b4826e492fc1762fceaf9726f41575ca0a4567864d3d235da874818de18039de quay.io/k8scsi/hostpathplugin:v1.2.0-rc5],SizeBytes:28761497,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7],SizeBytes:22909487,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/echoserver@sha256:e9ba514b896cdf559eef8788b66c2c3ee55f3572df617647b4b0d8b6bf81cf19 gcr.io/kubernetes-e2e-test-images/echoserver:2.2],SizeBytes:21692741,},ContainerImage{Names:[quay.io/k8scsi/mock-driver@sha256:e0eed916b7d970bad2b7d9875f9ad16932f987f0f3d91ec5d86da68b0b5cc9d1 quay.io/k8scsi/mock-driver:v2.1.0],SizeBytes:16226335,},ContainerImage{Names:[nginx@sha256:485b610fefec7ff6c463ced9623314a04ed67e3945b9c08d7e53a47f6d108dc7 nginx:1.14-alpine],SizeBytes:16032814,},ContainerImage{Names:[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0],SizeBytes:15815995,},ContainerImage{Names:[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0],SizeBytes:14967303,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/dnsutils@sha256:2abeee84efb79c14d731966e034af33bf324d3b26ca28497555511ff094b3ddd gcr.io/kubernetes-e2e-test-images/dnsutils:1.1],SizeBytes:9349974,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nonewprivs@sha256:10066e9039219449fe3c81f38fe01928f87914150768ab81b62a468e51fa7411 gcr.io/kubernetes-e2e-test-images/nonewprivs:1.0],SizeBytes:6757579,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nautilus@sha256:33a732d4c42a266912a5091598a0f07653c9134db4b8d571690d8afd509e0bfc gcr.io/kubernetes-e2e-test-images/nautilus:1.0],SizeBytes:4753501,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/kitten@sha256:bcbc4875c982ab39aa7c4f6acf4a287f604e996d9f34a3fbda8c3d1a7457d1f6 gcr.io/kubernetes-e2e-test-images/kitten:1.0],SizeBytes:4747037,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/test-webserver@sha256:7f93d6e32798ff28bc6289254d0c2867fe2c849c8e46edc50f8624734309812e gcr.io/kubernetes-e2e-test-images/test-webserver:1.0],SizeBytes:4732240,},ContainerImage{Names:[alpine@sha256:8421d9a84432575381bfabd248f1eb56f3aa21d9d7cd2511583c68c9b7511d10 alpine:3.7],SizeBytes:4206494,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 gcr.io/kubernetes-e2e-test-images/mounttest:1.0],SizeBytes:1563521,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest-user@sha256:17319ca525ee003681fccf7e8c6b1b910ff4f49b653d939ac7f9b6e7c463933d gcr.io/kubernetes-e2e-test-images/mounttest-user:1.0],SizeBytes:1450451,},ContainerImage{Names:[busybox@sha256:1303dbf110c57f3edf68d9f5a16c082ec06c4cf7604831669faf2c712260b5a0 busybox:latest],SizeBytes:1219790,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea k8s.gcr.io/pause-amd64:3.1 k8s.gcr.io/pause:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 20:57:13.024: INFO: Logging kubelet events for node k8s-agentpool-24944505-vmss000000 Nov 13 20:57:13.083: INFO: Logging pods the kubelet thinks is on node k8s-agentpool-24944505-vmss000000 Nov 13 20:57:13.225: INFO: csi-hostpathplugin-0 started at 2019-11-13 20:57:05 +0000 UTC (0+0 container statuses recorded) Nov 13 20:57:13.225: INFO: frontend-79ff456bff-5dv5l started at 2019-11-13 20:55:52 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.225: INFO: Container php-redis ready: false, restart count 0 Nov 13 20:57:13.225: INFO: redis-slave-68cd9c48b4-bgqzt started at 2019-11-13 20:55:54 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.225: INFO: Container slave ready: false, restart count 0 Nov 13 20:57:13.225: INFO: webserver-deployment-595b5b9587-wxmvq started at 2019-11-13 20:56:48 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.225: INFO: Container httpd ready: false, restart count 0 Nov 13 20:57:13.225: INFO: webserver-deployment-595b5b9587-vsg8d started at 2019-11-13 20:56:51 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.225: INFO: Container httpd ready: false, restart count 0 Nov 13 20:57:13.225: INFO: rs-pod1-885q4 started at 2019-11-13 20:56:22 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.225: INFO: Container pod1 ready: false, restart count 0 Nov 13 20:57:13.225: INFO: hostexec-k8s-agentpool-24944505-vmss000000 started at 2019-11-13 20:55:51 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.225: INFO: Container agnhost ready: true, restart count 0 Nov 13 20:57:13.225: INFO: security-context-e83563ef-f283-499d-8575-180e68014733 started at 2019-11-13 20:55:41 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.225: INFO: Container write-pod ready: true, restart count 0 Nov 13 20:57:13.225: INFO: rs-vwqbs started at 2019-11-13 20:55:44 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.225: INFO: Container busybox ready: false, restart count 0 Nov 13 20:57:13.225: INFO: hostexec-k8s-agentpool-24944505-vmss000000 started at 2019-11-13 20:55:23 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.225: INFO: Container agnhost ready: true, restart count 0 Nov 13 20:57:13.225: INFO: pod-configmaps-9cecc3f2-b3e2-4fc2-b72c-97d0de396ccd started at 2019-11-13 20:55:43 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.225: INFO: Container configmap-volume-test ready: true, restart count 0 Nov 13 20:57:13.225: INFO: csi-hostpath-attacher-0 started at 2019-11-13 20:55:56 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.225: INFO: Container csi-attacher ready: false, restart count 0 Nov 13 20:57:13.225: INFO: rs-pod1-5b5m7 started at 2019-11-13 20:56:24 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.225: INFO: Container pod1 ready: false, restart count 0 Nov 13 20:57:13.225: INFO: csi-hostpathplugin-0 started at 2019-11-13 20:56:40 +0000 UTC (0+3 container statuses recorded) Nov 13 20:57:13.225: INFO: Container hostpath ready: false, restart count 0 Nov 13 20:57:13.225: INFO: Container liveness-probe ready: false, restart count 0 Nov 13 20:57:13.225: INFO: Container node-driver-registrar ready: false, restart count 0 Nov 13 20:57:13.225: INFO: local-client started at 2019-11-13 20:56:59 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.225: INFO: Container local-client ready: false, restart count 0 Nov 13 20:57:13.225: INFO: security-context-dda17755-1b97-4e84-8d4e-f1f7d7ffbb9d started at 2019-11-13 20:56:24 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.225: INFO: Container write-pod ready: false, restart count 0 Nov 13 20:57:13.225: INFO: hostexec-k8s-agentpool-24944505-vmss000000 started at 2019-11-13 20:55:39 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.225: INFO: Container agnhost ready: true, restart count 0 Nov 13 20:57:13.225: INFO: netserver-1 started at 2019-11-13 20:56:58 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.225: INFO: Container webserver ready: false, restart count 0 Nov 13 20:57:13.225: INFO: netserver-0 started at 2019-11-13 20:57:11 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.225: INFO: Container webserver ready: false, restart count 0 Nov 13 20:57:13.225: INFO: rs-dbxb6 started at 2019-11-13 20:55:45 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.225: INFO: Container busybox ready: false, restart count 0 Nov 13 20:57:13.225: INFO: webserver-deployment-595b5b9587-r44nd started at 2019-11-13 20:56:47 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.225: INFO: Container httpd ready: false, restart count 0 Nov 13 20:57:13.225: INFO: redis-master-6ff87f4db7-h25v2 started at 2019-11-13 20:55:53 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.225: INFO: Container master ready: false, restart count 0 Nov 13 20:57:13.225: INFO: rs-pod1-4cp5c started at 2019-11-13 20:56:23 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.225: INFO: Container pod1 ready: false, restart count 0 Nov 13 20:57:13.225: INFO: webserver-deployment-595b5b9587-n4wcw started at 2019-11-13 20:56:50 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.225: INFO: Container httpd ready: false, restart count 0 Nov 13 20:57:13.225: INFO: metrics-server-58ff8c5ddf-r2d7f started at 2019-11-13 20:42:59 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.225: INFO: Container metrics-server ready: true, restart count 0 Nov 13 20:57:13.225: INFO: pfpod started at 2019-11-13 20:55:37 +0000 UTC (0+2 container statuses recorded) Nov 13 20:57:13.225: INFO: Container portforwardtester ready: false, restart count 0 Nov 13 20:57:13.225: INFO: Container readiness ready: false, restart count 0 Nov 13 20:57:13.225: INFO: rs-q8rb8 started at 2019-11-13 20:55:44 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.225: INFO: Container busybox ready: false, restart count 0 Nov 13 20:57:13.225: INFO: csi-hostpath-provisioner-0 started at 2019-11-13 20:55:57 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.225: INFO: Container csi-provisioner ready: false, restart count 0 Nov 13 20:57:13.225: INFO: rs-pod1-qt8hr started at 2019-11-13 20:56:24 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.225: INFO: Container pod1 ready: false, restart count 0 Nov 13 20:57:13.225: INFO: csi-hostpath-resizer-0 started at 2019-11-13 20:52:46 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.225: INFO: Container csi-resizer ready: true, restart count 0 Nov 13 20:57:13.225: INFO: azure-ip-masq-agent-qksp6 started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.225: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 20:57:13.225: INFO: tiller-deploy-7559b6b885-qxwkm started at 2019-11-13 20:42:59 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.225: INFO: Container tiller ready: true, restart count 0 Nov 13 20:57:13.225: INFO: frontend-79ff456bff-jjcgg started at 2019-11-13 20:55:52 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.225: INFO: Container php-redis ready: false, restart count 0 Nov 13 20:57:13.225: INFO: webserver-deployment-595b5b9587-f649w started at 2019-11-13 20:56:52 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.225: INFO: Container httpd ready: false, restart count 0 Nov 13 20:57:13.225: INFO: rs-sbm8q started at 2019-11-13 20:55:44 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.225: INFO: Container busybox ready: false, restart count 0 Nov 13 20:57:13.225: INFO: keyvault-flexvolume-sn47m started at 2019-11-13 20:42:59 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.225: INFO: Container keyvault-flexvolume ready: true, restart count 0 Nov 13 20:57:13.225: INFO: blobfuse-flexvol-installer-drhkj started at 2019-11-13 20:42:59 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.225: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 Nov 13 20:57:13.225: INFO: rs-pod1-ppppq started at 2019-11-13 20:56:23 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.225: INFO: Container pod1 ready: false, restart count 0 Nov 13 20:57:13.225: INFO: csi-hostpath-provisioner-0 started at 2019-11-13 20:52:46 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.225: INFO: Container csi-provisioner ready: true, restart count 0 Nov 13 20:57:13.225: INFO: local-injector started at 2019-11-13 20:56:34 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.225: INFO: Container local-injector ready: false, restart count 0 Nov 13 20:57:13.225: INFO: rs-6j6bb started at 2019-11-13 20:55:44 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.225: INFO: Container busybox ready: false, restart count 0 Nov 13 20:57:13.225: INFO: httpd started at 2019-11-13 20:55:17 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.225: INFO: Container httpd ready: true, restart count 0 Nov 13 20:57:13.225: INFO: hostexec-k8s-agentpool-24944505-vmss000000 started at 2019-11-13 20:55:06 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.225: INFO: Container agnhost ready: true, restart count 0 Nov 13 20:57:13.225: INFO: security-context-d91e91bd-a9e7-4879-8de2-ae9db079c244 started at 2019-11-13 20:56:24 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.225: INFO: Container write-pod ready: false, restart count 0 Nov 13 20:57:13.225: INFO: csi-hostpath-attacher-0 started at 2019-11-13 20:52:45 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.225: INFO: Container csi-attacher ready: true, restart count 0 Nov 13 20:57:13.225: INFO: kube-proxy-h9n4c started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.225: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 20:57:13.225: INFO: kubernetes-dashboard-65966766b9-lq4lm started at 2019-11-13 20:42:59 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.225: INFO: Container kubernetes-dashboard ready: true, restart count 0 Nov 13 20:57:13.225: INFO: hostexec-k8s-agentpool-24944505-vmss000000 started at 2019-11-13 20:56:40 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.225: INFO: Container agnhost ready: false, restart count 0 Nov 13 20:57:13.225: INFO: csi-snapshotter-0 started at 2019-11-13 20:52:46 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.225: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 13 20:57:13.225: INFO: csi-hostpath-resizer-0 started at 2019-11-13 20:55:57 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.225: INFO: Container csi-resizer ready: false, restart count 0 Nov 13 20:57:13.225: INFO: csi-snapshotter-0 started at 2019-11-13 20:55:57 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.225: INFO: Container csi-snapshotter ready: false, restart count 0 W1113 20:57:13.282860 92631 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 20:57:13.732: INFO: Latency metrics for node k8s-agentpool-24944505-vmss000000 Nov 13 20:57:13.732: INFO: Logging node info for node k8s-agentpool-24944505-vmss000001 Nov 13 20:57:13.790: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool-24944505-vmss000001 /api/v1/nodes/k8s-agentpool-24944505-vmss000001 da675d2e-0aad-4a8f-b94d-1311fb000563 26992 0 2019-11-13 20:42:24 +0000 UTC <nil> <nil> map[agentpool:agentpool beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool-24944505-vmss000001 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: storageprofile:managed storagetier:Premium_LRS] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-351":"k8s-agentpool-24944505-vmss000001","csi-hostpath-ephemeral-5584":"k8s-agentpool-24944505-vmss000001","csi-hostpath-provisioning-1638":"k8s-agentpool-24944505-vmss000001","csi-hostpath-provisioning-225":"k8s-agentpool-24944505-vmss000001","csi-hostpath-provisioning-7099":"k8s-agentpool-24944505-vmss000001","csi-hostpath-volume-6291":"k8s-agentpool-24944505-vmss000001","csi-hostpath-volume-expand-8458":"k8s-agentpool-24944505-vmss000001","csi-mock-csi-mock-volumes-597":"csi-mock-csi-mock-volumes-597","csi-mock-csi-mock-volumes-8812":"csi-mock-csi-mock-volumes-8812","csi-mock-csi-mock-volumes-9088":"csi-mock-csi-mock-volumes-9088"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-24944505-vmss/virtualMachines/1,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16797569024 0} {<nil>} 16403876Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16011137024 0} {<nil>} 15635876Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 20:56:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:24 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 20:56:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:24 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 20:56:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:24 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 20:56:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.248.0.5,},NodeAddress{Type:Hostname,Address:k8s-agentpool-24944505-vmss000001,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:6ed41b2c4b884fccaf94748594531e06,SystemUUID:9CE5C2E3-7C34-C64A-9734-2477E19937B7,BootID:306ac3f6-a23b-423e-83c9-e3d5f3938f6c,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[quay.io/kubernetes_incubator/nfs-provisioner@sha256:df762117e3c891f2d2ddff46ecb0776ba1f9f3c44cfd7739b0683bcd7a7954a8 quay.io/kubernetes_incubator/nfs-provisioner:v2.2.2],SizeBytes:391772778,},ContainerImage{Names:[gcr.io/google-samples/gb-frontend@sha256:35cb427341429fac3df10ff74600ea73e8ec0754d78f9ce89e0b4f3d70d53ba6 gcr.io/google-samples/gb-frontend:v6],SizeBytes:373099368,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/volume/nfs@sha256:c2ad734346f608a5f7d69cfded93c4e8094069320657bd372d12ba21dea3ea71 gcr.io/kubernetes-e2e-test-images/volume/nfs:1.0],SizeBytes:225358913,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/jessie-dnsutils@sha256:ad583e33cb284f7ef046673809b146ec4053cda19b54a85d2b180a86169715eb gcr.io/kubernetes-e2e-test-images/jessie-dnsutils:1.0],SizeBytes:195659796,},ContainerImage{Names:[httpd@sha256:addd70e4ee83f3bc9a4c1c7c41e37927ba47faf639312fc936df3afad7926f5a httpd:2.4.39-alpine],SizeBytes:126894770,},ContainerImage{Names:[httpd@sha256:eb8ccf084cf3e80eece1add239effefd171eb39adbc154d33c14260d905d4060 httpd:2.4.38-alpine],SizeBytes:123781643,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727 gcr.io/kubernetes-e2e-test-images/agnhost:2.6],SizeBytes:57345321,},ContainerImage{Names:[quay.io/k8scsi/csi-provisioner@sha256:0efcb424f1dde9b9fb11a1a14f2e48ab47e1c3f08bc3a929990dcfcb1f7ab34f quay.io/k8scsi/csi-provisioner:v1.4.0-rc1],SizeBytes:54431016,},ContainerImage{Names:[quay.io/k8scsi/csi-snapshotter@sha256:e3d3e742e32d00488fdb401045b9b1d033d7ca0ab6e760f77b24750fc95e5f70 quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1],SizeBytes:51703561,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0],SizeBytes:46226754,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:0aba670b4d9d6b2e720bbf575d733156c676b693ca26501235444490300db838 quay.io/k8scsi/csi-attacher:v1.1.0],SizeBytes:42839085,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:7d46fb6eb8b890dc546029d1565d502b4a1d974d33625c6ee2bc7991b77fc1a1 quay.io/k8scsi/csi-resizer:v0.2.0],SizeBytes:42817100,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:f315c9042e56def3c05c6b04fe79ec9da6d39ddc557ca365a76cf35964ea08b6 quay.io/k8scsi/csi-resizer:v0.1.0],SizeBytes:42623056,},ContainerImage{Names:[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0],SizeBytes:42488424,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nonroot@sha256:d4ede5c74517090b6686219059118ed178cf4620f5db8781b32f806bb1e7395b gcr.io/kubernetes-e2e-test-images/nonroot:1.0],SizeBytes:42321438,},ContainerImage{Names:[redis@sha256:50899ea1ceed33fa03232f3ac57578a424faa1742c1ac9c7a7bdb95cdf19b858 redis:5.0.5-alpine],SizeBytes:29331594,},ContainerImage{Names:[quay.io/k8scsi/hostpathplugin@sha256:b4826e492fc1762fceaf9726f41575ca0a4567864d3d235da874818de18039de quay.io/k8scsi/hostpathplugin:v1.2.0-rc5],SizeBytes:28761497,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7],SizeBytes:22909487,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/echoserver@sha256:e9ba514b896cdf559eef8788b66c2c3ee55f3572df617647b4b0d8b6bf81cf19 gcr.io/kubernetes-e2e-test-images/echoserver:2.2],SizeBytes:21692741,},ContainerImage{Names:[quay.io/k8scsi/mock-driver@sha256:e0eed916b7d970bad2b7d9875f9ad16932f987f0f3d91ec5d86da68b0b5cc9d1 quay.io/k8scsi/mock-driver:v2.1.0],SizeBytes:16226335,},ContainerImage{Names:[nginx@sha256:485b610fefec7ff6c463ced9623314a04ed67e3945b9c08d7e53a47f6d108dc7 nginx:1.14-alpine],SizeBytes:16032814,},ContainerImage{Names:[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0],SizeBytes:15815995,},ContainerImage{Names:[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0],SizeBytes:14967303,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/dnsutils@sha256:2abeee84efb79c14d731966e034af33bf324d3b26ca28497555511ff094b3ddd gcr.io/kubernetes-e2e-test-images/dnsutils:1.1],SizeBytes:9349974,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nautilus@sha256:33a732d4c42a266912a5091598a0f07653c9134db4b8d571690d8afd509e0bfc gcr.io/kubernetes-e2e-test-images/nautilus:1.0],SizeBytes:4753501,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/kitten@sha256:bcbc4875c982ab39aa7c4f6acf4a287f604e996d9f34a3fbda8c3d1a7457d1f6 gcr.io/kubernetes-e2e-test-images/kitten:1.0],SizeBytes:4747037,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/test-webserver@sha256:7f93d6e32798ff28bc6289254d0c2867fe2c849c8e46edc50f8624734309812e gcr.io/kubernetes-e2e-test-images/test-webserver:1.0],SizeBytes:4732240,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 gcr.io/kubernetes-e2e-test-images/mounttest:1.0],SizeBytes:1563521,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea k8s.gcr.io/pause-amd64:3.1 k8s.gcr.io/pause:3.1],SizeBytes:742472,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-8812^4],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-8812^4,DevicePath:,},},Config:nil,},} Nov 13 20:57:13.791: INFO: Logging kubelet events for node k8s-agentpool-24944505-vmss000001 Nov 13 20:57:13.850: INFO: Logging pods the kubelet thinks is on node k8s-agentpool-24944505-vmss000001 Nov 13 20:57:13.984: INFO: keyvault-flexvolume-7phpr started at 2019-11-13 20:42:58 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.984: INFO: Container keyvault-flexvolume ready: true, restart count 0 Nov 13 20:57:13.984: INFO: rs-jnh4t started at 2019-11-13 20:55:45 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.984: INFO: Container busybox ready: true, restart count 0 Nov 13 20:57:13.984: INFO: pod-subpath-test-local-preprovisionedpv-j79c started at 2019-11-13 20:57:04 +0000 UTC (1+2 container statuses recorded) Nov 13 20:57:13.984: INFO: Init container test-init-subpath-local-preprovisionedpv-j79c ready: false, restart count 0 Nov 13 20:57:13.984: INFO: Container test-container-subpath-local-preprovisionedpv-j79c ready: false, restart count 0 Nov 13 20:57:13.984: INFO: Container test-container-volume-local-preprovisionedpv-j79c ready: false, restart count 0 Nov 13 20:57:13.984: INFO: pod-projected-configmaps-c9772722-2110-46fd-98b7-395d39326281 started at 2019-11-13 20:56:49 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.984: INFO: Container projected-configmap-volume-test ready: false, restart count 0 Nov 13 20:57:13.984: INFO: pvc-tester-cbnwd started at 2019-11-13 20:56:50 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.984: INFO: Container write-pod ready: false, restart count 0 Nov 13 20:57:13.984: INFO: csi-hostpath-provisioner-0 started at 2019-11-13 20:50:19 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.984: INFO: Container csi-provisioner ready: true, restart count 0 Nov 13 20:57:13.984: INFO: redis-slave-68cd9c48b4-zt97v started at 2019-11-13 20:55:53 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.984: INFO: Container slave ready: true, restart count 0 Nov 13 20:57:13.984: INFO: csi-hostpath-resizer-0 started at 2019-11-13 20:50:19 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.984: INFO: Container csi-resizer ready: true, restart count 0 Nov 13 20:57:13.984: INFO: rs-hqk9w started at 2019-11-13 20:55:45 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.984: INFO: Container busybox ready: true, restart count 0 Nov 13 20:57:13.984: INFO: rs-vsz8b started at 2019-11-13 20:55:45 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.984: INFO: Container busybox ready: true, restart count 0 Nov 13 20:57:13.984: INFO: blobfuse-flexvol-installer-cpm6h started at 2019-11-13 20:42:58 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.984: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 Nov 13 20:57:13.984: INFO: webserver-deployment-595b5b9587-vgsjk started at 2019-11-13 20:56:45 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.984: INFO: Container httpd ready: false, restart count 0 Nov 13 20:57:13.984: INFO: frontend-79ff456bff-5ztnn started at 2019-11-13 20:55:52 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.984: INFO: Container php-redis ready: true, restart count 0 Nov 13 20:57:13.984: INFO: all-succeed-msl46 started at 2019-11-13 20:56:31 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.984: INFO: Container c ready: false, restart count 0 Nov 13 20:57:13.984: INFO: csi-mockplugin-attacher-0 started at 2019-11-13 20:53:56 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.984: INFO: Container csi-attacher ready: true, restart count 0 Nov 13 20:57:13.984: INFO: csi-hostpath-attacher-0 started at 2019-11-13 20:50:18 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.984: INFO: Container csi-attacher ready: true, restart count 0 Nov 13 20:57:13.984: INFO: failure-2-mgssf started at 2019-11-13 20:56:52 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.984: INFO: Container failure-2 ready: false, restart count 0 Nov 13 20:57:13.984: INFO: pod-subpath-test-local-preprovisionedpv-n8c6 started at 2019-11-13 20:57:06 +0000 UTC (1+2 container statuses recorded) Nov 13 20:57:13.984: INFO: Init container test-init-subpath-local-preprovisionedpv-n8c6 ready: false, restart count 0 Nov 13 20:57:13.984: INFO: Container test-container-subpath-local-preprovisionedpv-n8c6 ready: false, restart count 0 Nov 13 20:57:13.984: INFO: Container test-container-volume-local-preprovisionedpv-n8c6 ready: false, restart count 0 Nov 13 20:57:13.984: INFO: azure-ip-masq-agent-wwpg7 started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.984: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 20:57:13.984: INFO: webserver-deployment-595b5b9587-89zfs started at 2019-11-13 20:56:47 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.984: INFO: Container httpd ready: false, restart count 0 Nov 13 20:57:13.984: INFO: pod-prestop-hook-6a39099a-c80d-43f7-b126-406a4e28d257 started at 2019-11-13 20:56:36 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.984: INFO: Container nginx ready: false, restart count 0 Nov 13 20:57:13.984: INFO: rs-4vqjb started at 2019-11-13 20:55:45 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.984: INFO: Container busybox ready: true, restart count 0 Nov 13 20:57:13.984: INFO: hostexec-k8s-agentpool-24944505-vmss000001 started at 2019-11-13 20:56:22 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.984: INFO: Container agnhost ready: false, restart count 0 Nov 13 20:57:13.984: INFO: webserver-deployment-595b5b9587-z7pgw started at 2019-11-13 20:56:49 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.984: INFO: Container httpd ready: false, restart count 0 Nov 13 20:57:13.984: INFO: kube-proxy-7bpf7 started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.984: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 20:57:13.984: INFO: webserver-deployment-595b5b9587-2r8s4 started at 2019-11-13 20:56:45 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.984: INFO: Container httpd ready: false, restart count 0 Nov 13 20:57:13.984: INFO: failure-1 started at 2019-11-13 20:56:01 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.984: INFO: Container failure-1 ready: false, restart count 0 Nov 13 20:57:13.984: INFO: all-succeed-pgl44 started at 2019-11-13 20:56:31 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.984: INFO: Container c ready: false, restart count 0 Nov 13 20:57:13.984: INFO: netserver-0 started at 2019-11-13 20:56:58 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.984: INFO: Container webserver ready: false, restart count 0 Nov 13 20:57:13.984: INFO: nfs-server started at 2019-11-13 20:55:52 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.984: INFO: Container nfs-server ready: true, restart count 0 Nov 13 20:57:13.984: INFO: hostexec-k8s-agentpool-24944505-vmss000001 started at 2019-11-13 20:56:08 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.984: INFO: Container agnhost ready: true, restart count 0 Nov 13 20:57:13.984: INFO: inline-volume-tester-hblrb started at 2019-11-13 20:50:19 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.984: INFO: Container csi-volume-tester ready: false, restart count 0 Nov 13 20:57:13.984: INFO: csi-mockplugin-0 started at 2019-11-13 20:53:56 +0000 UTC (0+3 container statuses recorded) Nov 13 20:57:13.984: INFO: Container csi-provisioner ready: true, restart count 0 Nov 13 20:57:13.984: INFO: Container driver-registrar ready: true, restart count 0 Nov 13 20:57:13.984: INFO: Container mock ready: true, restart count 0 Nov 13 20:57:13.984: INFO: csi-hostpathplugin-0 started at 2019-11-13 20:55:41 +0000 UTC (0+3 container statuses recorded) Nov 13 20:57:13.984: INFO: Container hostpath ready: true, restart count 0 Nov 13 20:57:13.984: INFO: Container liveness-probe ready: true, restart count 0 Nov 13 20:57:13.984: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 13 20:57:13.984: INFO: rs-bzvqg started at 2019-11-13 20:55:45 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.984: INFO: Container busybox ready: true, restart count 0 Nov 13 20:57:13.984: INFO: webserver-deployment-595b5b9587-z64ln started at 2019-11-13 20:56:46 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.984: INFO: Container httpd ready: false, restart count 0 Nov 13 20:57:13.984: INFO: success started at 2019-11-13 20:55:42 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.984: INFO: Container success ready: false, restart count 0 Nov 13 20:57:13.984: INFO: netserver-1 started at 2019-11-13 20:57:12 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.984: INFO: Container webserver ready: false, restart count 0 Nov 13 20:57:13.984: INFO: pvc-volume-tester-vclb5 started at 2019-11-13 20:54:17 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.984: INFO: Container volume-tester ready: true, restart count 0 Nov 13 20:57:13.984: INFO: coredns-87f5d796-jd8bc started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.984: INFO: Container coredns ready: true, restart count 0 Nov 13 20:57:13.984: INFO: hostexec-k8s-agentpool-24944505-vmss000001 started at 2019-11-13 20:56:07 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.984: INFO: Container agnhost ready: true, restart count 0 Nov 13 20:57:13.984: INFO: csi-snapshotter-0 started at 2019-11-13 20:50:19 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.984: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 13 20:57:13.984: INFO: hostexec-k8s-agentpool-24944505-vmss000001 started at 2019-11-13 20:56:17 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.984: INFO: Container agnhost ready: true, restart count 0 Nov 13 20:57:13.984: INFO: send-events-b20e2ffa-1440-482b-a9a5-b7c7fdcfd253 started at 2019-11-13 20:56:42 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:13.984: INFO: Container p ready: false, restart count 0 W1113 20:57:14.044329 92631 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 20:57:14.437: INFO: Latency metrics for node k8s-agentpool-24944505-vmss000001 Nov 13 20:57:14.437: INFO: Logging node info for node k8s-master-24944505-vmss000000 Nov 13 20:57:14.492: INFO: Node Info: &Node{ObjectMeta:{k8s-master-24944505-vmss000000 /api/v1/nodes/k8s-master-24944505-vmss000000 048fbae2-b97f-43a2-b471-398180136f12 27055 0 2019-11-13 20:42:13 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-24944505-vmss000000 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-24944505-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284887552 0} {<nil>} 7114148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6498455552 0} {<nil>} 6346148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 20:57:06 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 20:57:06 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 20:57:06 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 20:57:06 +0000 UTC,LastTransitionTime:2019-11-13 20:42:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.4,},NodeAddress{Type:Hostname,Address:k8s-master-24944505-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:7bd754c0dc8647518c3043bcae3a76cc,SystemUUID:FA5CFE99-8148-6C46-B4E6-5EA2A5DE9D2A,BootID:74f7b0a2-e85c-438e-a642-c5b32900bb28,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:3237a53b135d8692ade82eca845fd0259ecbfcbe05fad35400e7d5c2018c4abd k8sprow.azurecr.io/azure-cloud-controller-manager:1194710255657816064-e688ecb0],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 20:57:14.492: INFO: Logging kubelet events for node k8s-master-24944505-vmss000000 Nov 13 20:57:14.557: INFO: Logging pods the kubelet thinks is on node k8s-master-24944505-vmss000000 Nov 13 20:57:14.643: INFO: kube-scheduler-k8s-master-24944505-vmss000000 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:14.643: INFO: Container kube-scheduler ready: true, restart count 0 Nov 13 20:57:14.643: INFO: azure-ip-masq-agent-6pnst started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:14.643: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 20:57:14.643: INFO: kube-proxy-d9jjh started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:14.643: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 20:57:14.643: INFO: cloud-controller-manager-k8s-master-24944505-vmss000000 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:14.643: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 13 20:57:14.643: INFO: kube-addon-manager-k8s-master-24944505-vmss000000 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:14.643: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 13 20:57:14.643: INFO: kube-apiserver-k8s-master-24944505-vmss000000 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:14.643: INFO: Container kube-apiserver ready: true, restart count 0 Nov 13 20:57:14.643: INFO: kube-controller-manager-k8s-master-24944505-vmss000000 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:14.643: INFO: Container kube-controller-manager ready: true, restart count 0 W1113 20:57:14.698886 92631 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 20:57:14.840: INFO: Latency metrics for node k8s-master-24944505-vmss000000 Nov 13 20:57:14.840: INFO: Logging node info for node k8s-master-24944505-vmss000001 Nov 13 20:57:14.896: INFO: Node Info: &Node{ObjectMeta:{k8s-master-24944505-vmss000001 /api/v1/nodes/k8s-master-24944505-vmss000001 a2619878-b119-44f9-b89f-4073bebbb56c 27012 0 2019-11-13 20:42:40 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-24944505-vmss000001 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.6.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-24944505-vmss/virtualMachines/1,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.6.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284883456 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6498451456 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:43:08 +0000 UTC,LastTransitionTime:2019-11-13 20:43:08 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 20:57:02 +0000 UTC,LastTransitionTime:2019-11-13 20:42:34 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 20:57:02 +0000 UTC,LastTransitionTime:2019-11-13 20:42:34 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 20:57:02 +0000 UTC,LastTransitionTime:2019-11-13 20:42:34 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 20:57:02 +0000 UTC,LastTransitionTime:2019-11-13 20:42:40 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.5,},NodeAddress{Type:Hostname,Address:k8s-master-24944505-vmss000001,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:c866a7c1ed64481990416e24387dd317,SystemUUID:EBE64557-2AC8-5740-9FFE-AB9F12412FBC,BootID:3dbb0e0e-0479-4821-bc3f-a8269d250e4c,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:3237a53b135d8692ade82eca845fd0259ecbfcbe05fad35400e7d5c2018c4abd k8sprow.azurecr.io/azure-cloud-controller-manager:1194710255657816064-e688ecb0],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 20:57:14.896: INFO: Logging kubelet events for node k8s-master-24944505-vmss000001 Nov 13 20:57:14.955: INFO: Logging pods the kubelet thinks is on node k8s-master-24944505-vmss000001 Nov 13 20:57:15.031: INFO: kube-scheduler-k8s-master-24944505-vmss000001 started at 2019-11-13 20:42:28 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:15.031: INFO: Container kube-scheduler ready: true, restart count 0 Nov 13 20:57:15.031: INFO: cloud-controller-manager-k8s-master-24944505-vmss000001 started at 2019-11-13 20:42:28 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:15.031: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 13 20:57:15.031: INFO: kube-addon-manager-k8s-master-24944505-vmss000001 started at 2019-11-13 20:42:28 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:15.031: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 13 20:57:15.031: INFO: kube-apiserver-k8s-master-24944505-vmss000001 started at 2019-11-13 20:42:28 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:15.031: INFO: Container kube-apiserver ready: true, restart count 0 Nov 13 20:57:15.031: INFO: kube-controller-manager-k8s-master-24944505-vmss000001 started at 2019-11-13 20:42:28 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:15.031: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 13 20:57:15.031: INFO: kube-proxy-w4bz5 started at 2019-11-13 20:42:41 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:15.031: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 20:57:15.031: INFO: azure-ip-masq-agent-bs52d started at 2019-11-13 20:42:41 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:15.031: INFO: Container azure-ip-masq-agent ready: true, restart count 0 W1113 20:57:15.096639 92631 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 20:57:15.244: INFO: Latency metrics for node k8s-master-24944505-vmss000001 Nov 13 20:57:15.244: INFO: Logging node info for node k8s-master-24944505-vmss000002 Nov 13 20:57:15.311: INFO: Node Info: &Node{ObjectMeta:{k8s-master-24944505-vmss000002 /api/v1/nodes/k8s-master-24944505-vmss000002 f3b05267-a009-4dba-9f97-4f772e48707e 27053 0 2019-11-13 20:42:13 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-24944505-vmss000002 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.4.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-24944505-vmss/virtualMachines/2,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284887552 0} {<nil>} 7114148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6498455552 0} {<nil>} 6346148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 20:57:06 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 20:57:06 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 20:57:06 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 20:57:06 +0000 UTC,LastTransitionTime:2019-11-13 20:42:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.6,},NodeAddress{Type:Hostname,Address:k8s-master-24944505-vmss000002,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:fc1777f37ced4b9c97acb9b30e276c4f,SystemUUID:E9DF19DC-D02C-714C-A7D6-0BA6CE959C55,BootID:622958a8-d45d-4edf-9266-4dbb759e2359,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:3237a53b135d8692ade82eca845fd0259ecbfcbe05fad35400e7d5c2018c4abd k8sprow.azurecr.io/azure-cloud-controller-manager:1194710255657816064-e688ecb0],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 20:57:15.311: INFO: Logging kubelet events for node k8s-master-24944505-vmss000002 Nov 13 20:57:15.370: INFO: Logging pods the kubelet thinks is on node k8s-master-24944505-vmss000002 Nov 13 20:57:15.452: INFO: azure-ip-masq-agent-6skf2 started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:15.452: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 20:57:15.452: INFO: kube-proxy-xwz78 started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:15.452: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 20:57:15.452: INFO: kube-addon-manager-k8s-master-24944505-vmss000002 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:15.452: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 13 20:57:15.452: INFO: kube-apiserver-k8s-master-24944505-vmss000002 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:15.452: INFO: Container kube-apiserver ready: true, restart count 0 Nov 13 20:57:15.452: INFO: kube-controller-manager-k8s-master-24944505-vmss000002 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:15.452: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 13 20:57:15.452: INFO: kube-scheduler-k8s-master-24944505-vmss000002 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:15.452: INFO: Container kube-scheduler ready: true, restart count 0 Nov 13 20:57:15.452: INFO: cloud-controller-manager-k8s-master-24944505-vmss000002 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:15.452: INFO: Container cloud-controller-manager ready: true, restart count 0 W1113 20:57:15.516231 92631 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 20:57:15.687: INFO: Latency metrics for node k8s-master-24944505-vmss000002 Nov 13 20:57:15.687: INFO: Logging node info for node k8s-master-24944505-vmss000003 Nov 13 20:57:15.742: INFO: Node Info: &Node{ObjectMeta:{k8s-master-24944505-vmss000003 /api/v1/nodes/k8s-master-24944505-vmss000003 888cc19e-c62d-4945-a425-97d14e38a0b8 26982 0 2019-11-13 20:42:35 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-24944505-vmss000003 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.5.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-24944505-vmss/virtualMachines/3,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.5.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284887552 0} {<nil>} 7114148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6498455552 0} {<nil>} 6346148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 20:56:57 +0000 UTC,LastTransitionTime:2019-11-13 20:42:29 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 20:56:57 +0000 UTC,LastTransitionTime:2019-11-13 20:42:29 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 20:56:57 +0000 UTC,LastTransitionTime:2019-11-13 20:42:29 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 20:56:57 +0000 UTC,LastTransitionTime:2019-11-13 20:42:35 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.7,},NodeAddress{Type:Hostname,Address:k8s-master-24944505-vmss000003,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:2da5fe5a442b4e178a2b62ba91366e4b,SystemUUID:D2D1E816-D70E-3D40-B959-B00A2B76A83A,BootID:c5929aa1-3dac-45d5-bb2c-174c73c7300e,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:3237a53b135d8692ade82eca845fd0259ecbfcbe05fad35400e7d5c2018c4abd k8sprow.azurecr.io/azure-cloud-controller-manager:1194710255657816064-e688ecb0],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 20:57:15.742: INFO: Logging kubelet events for node k8s-master-24944505-vmss000003 Nov 13 20:57:15.802: INFO: Logging pods the kubelet thinks is on node k8s-master-24944505-vmss000003 Nov 13 20:57:15.883: INFO: kube-addon-manager-k8s-master-24944505-vmss000003 started at 2019-11-13 20:42:22 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:15.883: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 13 20:57:15.883: INFO: kube-apiserver-k8s-master-24944505-vmss000003 started at 2019-11-13 20:42:23 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:15.883: INFO: Container kube-apiserver ready: true, restart count 0 Nov 13 20:57:15.883: INFO: kube-controller-manager-k8s-master-24944505-vmss000003 started at 2019-11-13 20:42:23 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:15.883: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 13 20:57:15.883: INFO: kube-scheduler-k8s-master-24944505-vmss000003 started at 2019-11-13 20:42:23 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:15.883: INFO: Container kube-scheduler ready: true, restart count 0 Nov 13 20:57:15.883: INFO: cloud-controller-manager-k8s-master-24944505-vmss000003 started at 2019-11-13 20:42:44 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:15.883: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 13 20:57:15.883: INFO: azure-ip-masq-agent-nlqss started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:15.883: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 20:57:15.883: INFO: kube-proxy-kgh5h started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:15.883: INFO: Container kube-proxy ready: true, restart count 0 W1113 20:57:15.942339 92631 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 20:57:16.093: INFO: Latency metrics for node k8s-master-24944505-vmss000003 Nov 13 20:57:16.093: INFO: Logging node info for node k8s-master-24944505-vmss000004 Nov 13 20:57:16.148: INFO: Node Info: &Node{ObjectMeta:{k8s-master-24944505-vmss000004 /api/v1/nodes/k8s-master-24944505-vmss000004 a26d844f-0612-406b-9a65-7e2ac952a2d5 27061 0 2019-11-13 20:42:14 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-24944505-vmss000004 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-24944505-vmss/virtualMachines/4,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284887552 0} {<nil>} 7114148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6498455552 0} {<nil>} 6346148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 20:57:06 +0000 UTC,LastTransitionTime:2019-11-13 20:42:09 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 20:57:06 +0000 UTC,LastTransitionTime:2019-11-13 20:42:09 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 20:57:06 +0000 UTC,LastTransitionTime:2019-11-13 20:42:09 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 20:57:06 +0000 UTC,LastTransitionTime:2019-11-13 20:42:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.8,},NodeAddress{Type:Hostname,Address:k8s-master-24944505-vmss000004,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:2208959aca1045d9b48fce8d784c1e41,SystemUUID:E253DB3A-3A4D-5648-8169-B2E2976CFB4B,BootID:09bc6101-2da9-4c8e-8274-43db8c5fe1cf,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:3237a53b135d8692ade82eca845fd0259ecbfcbe05fad35400e7d5c2018c4abd k8sprow.azurecr.io/azure-cloud-controller-manager:1194710255657816064-e688ecb0],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 20:57:16.148: INFO: Logging kubelet events for node k8s-master-24944505-vmss000004 Nov 13 20:57:16.227: INFO: Logging pods the kubelet thinks is on node k8s-master-24944505-vmss000004 Nov 13 20:57:16.310: INFO: cloud-controller-manager-k8s-master-24944505-vmss000004 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:16.310: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 13 20:57:16.310: INFO: kube-addon-manager-k8s-master-24944505-vmss000004 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:16.310: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 13 20:57:16.310: INFO: kube-apiserver-k8s-master-24944505-vmss000004 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:16.310: INFO: Container kube-apiserver ready: true, restart count 0 Nov 13 20:57:16.310: INFO: kube-controller-manager-k8s-master-24944505-vmss000004 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:16.310: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 13 20:57:16.310: INFO: kube-scheduler-k8s-master-24944505-vmss000004 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:16.310: INFO: Container kube-scheduler ready: true, restart count 0 Nov 13 20:57:16.310: INFO: azure-ip-masq-agent-xnw2h started at 2019-11-13 20:42:38 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:16.310: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 20:57:16.310: INFO: kube-proxy-fld6p started at 2019-11-13 20:42:38 +0000 UTC (0+1 container statuses recorded) Nov 13 20:57:16.310: INFO: Container kube-proxy ready: true, restart count 0 W1113 20:57:16.376578 92631 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 20:57:16.502: INFO: Latency metrics for node k8s-master-24944505-vmss000004 Nov 13 20:57:16.502: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "sched-preemption-path-7365" for this suite. Nov 13 20:57:36.731: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered Nov 13 20:57:38.584: INFO: namespace sched-preemption-path-7365 deletion completed in 22.024591664s [AfterEach] [sig-scheduling] PreemptionExecutionPath test/e2e/scheduling/preemption.go:274 Nov 13 20:57:38.642: INFO: List existing priorities: Nov 13 20:57:38.642: INFO: p1/1 created at 2019-11-13 20:56:12 +0000 UTC Nov 13 20:57:38.642: INFO: p2/2 created at 2019-11-13 20:56:12 +0000 UTC Nov 13 20:57:38.642: INFO: p3/3 created at 2019-11-13 20:56:12 +0000 UTC Nov 13 20:57:38.642: INFO: p4/4 created at 2019-11-13 20:56:12 +0000 UTC Nov 13 20:57:38.642: INFO: system-cluster-critical/2000000000 created at 2019-11-13 20:42:14 +0000 UTC Nov 13 20:57:38.642: INFO: system-node-critical/2000001000 created at 2019-11-13 20:42:14 +0000 UTC
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[sig\-scheduling\]\sPreemptionExecutionPath\sruns\sReplicaSets\sto\sverify\spreemption\srunning\spath$'
test/e2e/scheduling/preemption.go:345 Nov 13 20:59:10.424: Unexpected error: <*errors.errorString | 0xc00237ea80>: { s: "replicaset \"rs-pod1\" never had desired number of .status.availableReplicas", } replicaset "rs-pod1" never had desired number of .status.availableReplicas occurred test/e2e/scheduling/preemption.go:510from junit_06.xml
[BeforeEach] [sig-scheduling] PreemptionExecutionPath test/e2e/framework/framework.go:151 �[1mSTEP�[0m: Creating a kubernetes client Nov 13 20:57:38.990: INFO: >>> kubeConfig: /workspace/aks880331199/kubeconfig/kubeconfig.westus2.json �[1mSTEP�[0m: Building a namespace api object, basename sched-preemption-path �[1mSTEP�[0m: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in sched-preemption-path-5324 �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace [BeforeEach] [sig-scheduling] PreemptionExecutionPath test/e2e/scheduling/preemption.go:302 �[1mSTEP�[0m: Finding an available node �[1mSTEP�[0m: Trying to launch a pod without a label to get a node which can launch it. �[1mSTEP�[0m: Explicitly delete pod here to free the resource it takes. Nov 13 20:58:09.797: INFO: found a healthy node: k8s-agentpool-24944505-vmss000001 [It] runs ReplicaSets to verify preemption running path test/e2e/scheduling/preemption.go:345 Nov 13 20:59:10.424: FAIL: Unexpected error: <*errors.errorString | 0xc00237ea80>: { s: "replicaset \"rs-pod1\" never had desired number of .status.availableReplicas", } replicaset "rs-pod1" never had desired number of .status.availableReplicas occurred [AfterEach] [sig-scheduling] PreemptionExecutionPath test/e2e/framework/framework.go:152 �[1mSTEP�[0m: Collecting events from namespace "sched-preemption-path-5324". �[1mSTEP�[0m: Found 34 events. Nov 13 20:59:10.532: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-7hl8k: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 13 20:59:10.532: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-7hl8k: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 13 20:59:10.532: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-7hl8k: {default-scheduler } Scheduled: Successfully assigned sched-preemption-path-5324/rs-pod1-7hl8k to k8s-agentpool-24944505-vmss000001 Nov 13 20:59:10.532: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-7j6k4: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 13 20:59:10.532: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-7j6k4: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 13 20:59:10.532: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-7j6k4: {default-scheduler } Scheduled: Successfully assigned sched-preemption-path-5324/rs-pod1-7j6k4 to k8s-agentpool-24944505-vmss000001 Nov 13 20:59:10.532: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-cnmm6: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 13 20:59:10.532: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-cnmm6: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 13 20:59:10.532: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-cnmm6: {default-scheduler } Scheduled: Successfully assigned sched-preemption-path-5324/rs-pod1-cnmm6 to k8s-agentpool-24944505-vmss000001 Nov 13 20:59:10.532: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-hzx6b: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 13 20:59:10.532: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-hzx6b: {default-scheduler } Scheduled: Successfully assigned sched-preemption-path-5324/rs-pod1-hzx6b to k8s-agentpool-24944505-vmss000001 Nov 13 20:59:10.532: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-hzx6b: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 13 20:59:10.532: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-xsqgc: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 13 20:59:10.532: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-xsqgc: {default-scheduler } Scheduled: Successfully assigned sched-preemption-path-5324/rs-pod1-xsqgc to k8s-agentpool-24944505-vmss000001 Nov 13 20:59:10.532: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for rs-pod1-xsqgc: {default-scheduler } FailedScheduling: 0/7 nodes are available: 6 node(s) didn't match node selector, 7 Insufficient example.com/fakecpu. Nov 13 20:59:10.533: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for without-label: {default-scheduler } Scheduled: Successfully assigned sched-preemption-path-5324/without-label to k8s-agentpool-24944505-vmss000001 Nov 13 20:59:10.533: INFO: At 2019-11-13 20:57:53 +0000 UTC - event for without-label: {kubelet k8s-agentpool-24944505-vmss000001} Pulling: Pulling image "k8s.gcr.io/pause:3.1" Nov 13 20:59:10.533: INFO: At 2019-11-13 20:57:53 +0000 UTC - event for without-label: {kubelet k8s-agentpool-24944505-vmss000001} Pulled: Successfully pulled image "k8s.gcr.io/pause:3.1" Nov 13 20:59:10.533: INFO: At 2019-11-13 20:57:59 +0000 UTC - event for without-label: {kubelet k8s-agentpool-24944505-vmss000001} Created: Created container without-label Nov 13 20:59:10.533: INFO: At 2019-11-13 20:58:04 +0000 UTC - event for without-label: {kubelet k8s-agentpool-24944505-vmss000001} Started: Started container without-label Nov 13 20:59:10.533: INFO: At 2019-11-13 20:58:09 +0000 UTC - event for without-label: {kubelet k8s-agentpool-24944505-vmss000001} Killing: Stopping container without-label Nov 13 20:59:10.533: INFO: At 2019-11-13 20:58:10 +0000 UTC - event for rs-pod1: {replicaset-controller } SuccessfulCreate: Created pod: rs-pod1-7hl8k Nov 13 20:59:10.533: INFO: At 2019-11-13 20:58:10 +0000 UTC - event for rs-pod1: {replicaset-controller } SuccessfulCreate: Created pod: rs-pod1-hzx6b Nov 13 20:59:10.533: INFO: At 2019-11-13 20:58:10 +0000 UTC - event for rs-pod1: {replicaset-controller } SuccessfulCreate: Created pod: rs-pod1-cnmm6 Nov 13 20:59:10.533: INFO: At 2019-11-13 20:58:10 +0000 UTC - event for rs-pod1: {replicaset-controller } SuccessfulCreate: Created pod: rs-pod1-xsqgc Nov 13 20:59:10.533: INFO: At 2019-11-13 20:58:10 +0000 UTC - event for rs-pod1: {replicaset-controller } SuccessfulCreate: Created pod: rs-pod1-7j6k4 Nov 13 20:59:10.533: INFO: At 2019-11-13 20:59:00 +0000 UTC - event for rs-pod1-hzx6b: {kubelet k8s-agentpool-24944505-vmss000001} Pulling: Pulling image "k8s.gcr.io/pause:3.1" Nov 13 20:59:10.533: INFO: At 2019-11-13 20:59:01 +0000 UTC - event for rs-pod1-hzx6b: {kubelet k8s-agentpool-24944505-vmss000001} Pulled: Successfully pulled image "k8s.gcr.io/pause:3.1" Nov 13 20:59:10.533: INFO: At 2019-11-13 20:59:03 +0000 UTC - event for rs-pod1-cnmm6: {kubelet k8s-agentpool-24944505-vmss000001} Pulled: Successfully pulled image "k8s.gcr.io/pause:3.1" Nov 13 20:59:10.533: INFO: At 2019-11-13 20:59:03 +0000 UTC - event for rs-pod1-cnmm6: {kubelet k8s-agentpool-24944505-vmss000001} Pulling: Pulling image "k8s.gcr.io/pause:3.1" Nov 13 20:59:10.533: INFO: At 2019-11-13 20:59:05 +0000 UTC - event for rs-pod1-hzx6b: {kubelet k8s-agentpool-24944505-vmss000001} Created: Created container pod1 Nov 13 20:59:10.533: INFO: At 2019-11-13 20:59:06 +0000 UTC - event for rs-pod1-cnmm6: {kubelet k8s-agentpool-24944505-vmss000001} Created: Created container pod1 Nov 13 20:59:10.533: INFO: At 2019-11-13 20:59:07 +0000 UTC - event for rs-pod1-cnmm6: {kubelet k8s-agentpool-24944505-vmss000001} Started: Started container pod1 Nov 13 20:59:10.533: INFO: At 2019-11-13 20:59:07 +0000 UTC - event for rs-pod1-hzx6b: {kubelet k8s-agentpool-24944505-vmss000001} Started: Started container pod1 Nov 13 20:59:10.591: INFO: POD NODE PHASE GRACE CONDITIONS Nov 13 20:59:10.591: INFO: rs-pod1-7hl8k k8s-agentpool-24944505-vmss000001 Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:58:29 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:58:29 +0000 UTC ContainersNotReady containers with unready status: [pod1]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:58:29 +0000 UTC ContainersNotReady containers with unready status: [pod1]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:58:23 +0000 UTC }] Nov 13 20:59:10.591: INFO: rs-pod1-7j6k4 k8s-agentpool-24944505-vmss000001 Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:58:27 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:58:27 +0000 UTC ContainersNotReady containers with unready status: [pod1]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:58:27 +0000 UTC ContainersNotReady containers with unready status: [pod1]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:58:23 +0000 UTC }] Nov 13 20:59:10.591: INFO: rs-pod1-cnmm6 k8s-agentpool-24944505-vmss000001 Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:58:24 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:58:24 +0000 UTC ContainersNotReady containers with unready status: [pod1]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:58:24 +0000 UTC ContainersNotReady containers with unready status: [pod1]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:58:23 +0000 UTC }] Nov 13 20:59:10.591: INFO: rs-pod1-hzx6b k8s-agentpool-24944505-vmss000001 Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:58:25 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:58:25 +0000 UTC ContainersNotReady containers with unready status: [pod1]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:58:25 +0000 UTC ContainersNotReady containers with unready status: [pod1]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:58:23 +0000 UTC }] Nov 13 20:59:10.591: INFO: rs-pod1-xsqgc k8s-agentpool-24944505-vmss000001 Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:58:27 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:58:27 +0000 UTC ContainersNotReady containers with unready status: [pod1]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:58:27 +0000 UTC ContainersNotReady containers with unready status: [pod1]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:58:23 +0000 UTC }] Nov 13 20:59:10.591: INFO: Nov 13 20:59:10.755: INFO: Logging node info for node k8s-agentpool-24944505-vmss000000 Nov 13 20:59:10.811: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool-24944505-vmss000000 /api/v1/nodes/k8s-agentpool-24944505-vmss000000 5bc7cf9e-80da-4e26-a2c2-20757e870d68 28336 0 2019-11-13 20:42:13 +0000 UTC <nil> <nil> map[agentpool:agentpool beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool-24944505-vmss000000 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: storageprofile:managed storagetier:Premium_LRS] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-1137":"k8s-agentpool-24944505-vmss000000","csi-hostpath-ephemeral-9135":"k8s-agentpool-24944505-vmss000000","csi-hostpath-provisioning-2585":"k8s-agentpool-24944505-vmss000000","csi-hostpath-provisioning-716":"k8s-agentpool-24944505-vmss000000","csi-hostpath-volume-expand-4561":"k8s-agentpool-24944505-vmss000000","csi-hostpath-volume-expand-9975":"k8s-agentpool-24944505-vmss000000","csi-mock-csi-mock-volumes-2863":"csi-mock-csi-mock-volumes-2863","csi-mock-csi-mock-volumes-5360":"csi-mock-csi-mock-volumes-5360","csi-mock-csi-mock-volumes-8606":"csi-mock-csi-mock-volumes-8606"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-24944505-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16797569024 0} {<nil>} 16403876Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16011137024 0} {<nil>} 15635876Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 20:58:17 +0000 UTC,LastTransitionTime:2019-11-13 20:42:03 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 20:58:17 +0000 UTC,LastTransitionTime:2019-11-13 20:42:03 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 20:58:17 +0000 UTC,LastTransitionTime:2019-11-13 20:42:03 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 20:58:17 +0000 UTC,LastTransitionTime:2019-11-13 20:42:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.248.0.4,},NodeAddress{Type:Hostname,Address:k8s-agentpool-24944505-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:0a2e8938ebfc440180ed5c0b5a9edbe7,SystemUUID:8075A1F3-D764-7248-9570-A764E51D7FF2,BootID:4f2b3cbe-c61c-4c6c-93ab-18fca0df5bc5,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[quay.io/kubernetes_incubator/nfs-provisioner@sha256:df762117e3c891f2d2ddff46ecb0776ba1f9f3c44cfd7739b0683bcd7a7954a8 quay.io/kubernetes_incubator/nfs-provisioner:v2.2.2],SizeBytes:391772778,},ContainerImage{Names:[gcr.io/google-samples/gb-frontend@sha256:35cb427341429fac3df10ff74600ea73e8ec0754d78f9ce89e0b4f3d70d53ba6 gcr.io/google-samples/gb-frontend:v6],SizeBytes:373099368,},ContainerImage{Names:[k8s.gcr.io/etcd@sha256:12c2c5e5731c3bcd56e6f1c05c0f9198b6f06793fa7fca2fb43aab9622dc4afa k8s.gcr.io/etcd:3.3.15],SizeBytes:246640776,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/volume/nfs@sha256:c2ad734346f608a5f7d69cfded93c4e8094069320657bd372d12ba21dea3ea71 gcr.io/kubernetes-e2e-test-images/volume/nfs:1.0],SizeBytes:225358913,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/jessie-dnsutils@sha256:ad583e33cb284f7ef046673809b146ec4053cda19b54a85d2b180a86169715eb gcr.io/kubernetes-e2e-test-images/jessie-dnsutils:1.0],SizeBytes:195659796,},ContainerImage{Names:[httpd@sha256:addd70e4ee83f3bc9a4c1c7c41e37927ba47faf639312fc936df3afad7926f5a httpd:2.4.39-alpine],SizeBytes:126894770,},ContainerImage{Names:[httpd@sha256:eb8ccf084cf3e80eece1add239effefd171eb39adbc154d33c14260d905d4060 httpd:2.4.38-alpine],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1],SizeBytes:121711221,},ContainerImage{Names:[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0],SizeBytes:71821984,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/sample-apiserver@sha256:1bafcc6fb1aa990b487850adba9cadc020e42d7905aa8a30481182a477ba24b0 gcr.io/kubernetes-e2e-test-images/sample-apiserver:1.10],SizeBytes:61365829,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727 gcr.io/kubernetes-e2e-test-images/agnhost:2.6],SizeBytes:57345321,},ContainerImage{Names:[quay.io/k8scsi/csi-provisioner@sha256:0efcb424f1dde9b9fb11a1a14f2e48ab47e1c3f08bc3a929990dcfcb1f7ab34f quay.io/k8scsi/csi-provisioner:v1.4.0-rc1],SizeBytes:54431016,},ContainerImage{Names:[quay.io/k8scsi/csi-snapshotter@sha256:e3d3e742e32d00488fdb401045b9b1d033d7ca0ab6e760f77b24750fc95e5f70 quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1],SizeBytes:51703561,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0],SizeBytes:46226754,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:0aba670b4d9d6b2e720bbf575d733156c676b693ca26501235444490300db838 quay.io/k8scsi/csi-attacher:v1.1.0],SizeBytes:42839085,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:7d46fb6eb8b890dc546029d1565d502b4a1d974d33625c6ee2bc7991b77fc1a1 quay.io/k8scsi/csi-resizer:v0.2.0],SizeBytes:42817100,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:f315c9042e56def3c05c6b04fe79ec9da6d39ddc557ca365a76cf35964ea08b6 quay.io/k8scsi/csi-resizer:v0.1.0],SizeBytes:42623056,},ContainerImage{Names:[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1],SizeBytes:42541759,},ContainerImage{Names:[gcr.io/google-containers/debian-base@sha256:6966a0aedd7592c18ff2dd803c08bd85780ee19f5e3a2e7cf908a4cd837afcde gcr.io/google-containers/debian-base:0.4.1],SizeBytes:42323657,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nonroot@sha256:d4ede5c74517090b6686219059118ed178cf4620f5db8781b32f806bb1e7395b gcr.io/kubernetes-e2e-test-images/nonroot:1.0],SizeBytes:42321438,},ContainerImage{Names:[redis@sha256:50899ea1ceed33fa03232f3ac57578a424faa1742c1ac9c7a7bdb95cdf19b858 redis:5.0.5-alpine],SizeBytes:29331594,},ContainerImage{Names:[quay.io/k8scsi/hostpathplugin@sha256:b4826e492fc1762fceaf9726f41575ca0a4567864d3d235da874818de18039de quay.io/k8scsi/hostpathplugin:v1.2.0-rc5],SizeBytes:28761497,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7],SizeBytes:22909487,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/echoserver@sha256:e9ba514b896cdf559eef8788b66c2c3ee55f3572df617647b4b0d8b6bf81cf19 gcr.io/kubernetes-e2e-test-images/echoserver:2.2],SizeBytes:21692741,},ContainerImage{Names:[quay.io/k8scsi/mock-driver@sha256:e0eed916b7d970bad2b7d9875f9ad16932f987f0f3d91ec5d86da68b0b5cc9d1 quay.io/k8scsi/mock-driver:v2.1.0],SizeBytes:16226335,},ContainerImage{Names:[nginx@sha256:485b610fefec7ff6c463ced9623314a04ed67e3945b9c08d7e53a47f6d108dc7 nginx:1.14-alpine],SizeBytes:16032814,},ContainerImage{Names:[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0],SizeBytes:15815995,},ContainerImage{Names:[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0],SizeBytes:14967303,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/dnsutils@sha256:2abeee84efb79c14d731966e034af33bf324d3b26ca28497555511ff094b3ddd gcr.io/kubernetes-e2e-test-images/dnsutils:1.1],SizeBytes:9349974,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nonewprivs@sha256:10066e9039219449fe3c81f38fe01928f87914150768ab81b62a468e51fa7411 gcr.io/kubernetes-e2e-test-images/nonewprivs:1.0],SizeBytes:6757579,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nautilus@sha256:33a732d4c42a266912a5091598a0f07653c9134db4b8d571690d8afd509e0bfc gcr.io/kubernetes-e2e-test-images/nautilus:1.0],SizeBytes:4753501,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/kitten@sha256:bcbc4875c982ab39aa7c4f6acf4a287f604e996d9f34a3fbda8c3d1a7457d1f6 gcr.io/kubernetes-e2e-test-images/kitten:1.0],SizeBytes:4747037,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/test-webserver@sha256:7f93d6e32798ff28bc6289254d0c2867fe2c849c8e46edc50f8624734309812e gcr.io/kubernetes-e2e-test-images/test-webserver:1.0],SizeBytes:4732240,},ContainerImage{Names:[alpine@sha256:8421d9a84432575381bfabd248f1eb56f3aa21d9d7cd2511583c68c9b7511d10 alpine:3.7],SizeBytes:4206494,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 gcr.io/kubernetes-e2e-test-images/mounttest:1.0],SizeBytes:1563521,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest-user@sha256:17319ca525ee003681fccf7e8c6b1b910ff4f49b653d939ac7f9b6e7c463933d gcr.io/kubernetes-e2e-test-images/mounttest-user:1.0],SizeBytes:1450451,},ContainerImage{Names:[busybox@sha256:1303dbf110c57f3edf68d9f5a16c082ec06c4cf7604831669faf2c712260b5a0 busybox:latest],SizeBytes:1219790,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea k8s.gcr.io/pause-amd64:3.1 k8s.gcr.io/pause:3.1],SizeBytes:742472,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-provisioning-2585^4a8a88a2-0658-11ea-9e95-000d3ac2cd71],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-provisioning-2585^4a8a88a2-0658-11ea-9e95-000d3ac2cd71,DevicePath:,},},Config:nil,},} Nov 13 20:59:10.812: INFO: Logging kubelet events for node k8s-agentpool-24944505-vmss000000 Nov 13 20:59:10.870: INFO: Logging pods the kubelet thinks is on node k8s-agentpool-24944505-vmss000000 Nov 13 20:59:11.014: INFO: azure-ip-masq-agent-qksp6 started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:11.014: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 20:59:11.014: INFO: tiller-deploy-7559b6b885-qxwkm started at 2019-11-13 20:42:59 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:11.014: INFO: Container tiller ready: true, restart count 0 Nov 13 20:59:11.014: INFO: csi-hostpath-provisioner-0 started at 2019-11-13 20:55:57 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:11.014: INFO: Container csi-provisioner ready: true, restart count 0 Nov 13 20:59:11.014: INFO: pod-secrets-ed86e8c4-95c6-48e6-ad00-e4fe31dcd12c started at 2019-11-13 20:58:44 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:11.014: INFO: Container secret-volume-test ready: false, restart count 0 Nov 13 20:59:11.015: INFO: pod-projected-configmaps-27a0ac86-29f0-4368-ab0b-8fcb710bc618 started at 2019-11-13 20:58:53 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:11.015: INFO: Container projected-configmap-volume-test ready: false, restart count 0 Nov 13 20:59:11.015: INFO: csi-mockplugin-attacher-0 started at 2019-11-13 20:58:59 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:11.015: INFO: Container csi-attacher ready: false, restart count 0 Nov 13 20:59:11.015: INFO: csi-mockplugin-resizer-0 started at 2019-11-13 20:59:00 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:11.015: INFO: Container csi-resizer ready: false, restart count 0 Nov 13 20:59:11.015: INFO: netserver-0 started at 2019-11-13 20:57:53 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:11.015: INFO: Container webserver ready: true, restart count 0 Nov 13 20:59:11.015: INFO: csi-snapshotter-0 started at 2019-11-13 20:58:52 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:11.015: INFO: Container csi-snapshotter ready: false, restart count 0 Nov 13 20:59:11.015: INFO: hostexec-k8s-agentpool-24944505-vmss000000 started at 2019-11-13 20:55:06 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:11.015: INFO: Container agnhost ready: true, restart count 0 Nov 13 20:59:11.015: INFO: pod-subpath-test-csi-hostpath-dynamicpv-cqgt started at 2019-11-13 20:58:08 +0000 UTC (0+2 container statuses recorded) Nov 13 20:59:11.015: INFO: Container test-container-subpath-csi-hostpath-dynamicpv-cqgt ready: false, restart count 0 Nov 13 20:59:11.015: INFO: Container test-container-volume-csi-hostpath-dynamicpv-cqgt ready: false, restart count 0 Nov 13 20:59:11.015: INFO: csi-mockplugin-0 started at 2019-11-13 20:58:58 +0000 UTC (0+3 container statuses recorded) Nov 13 20:59:11.015: INFO: Container csi-provisioner ready: false, restart count 0 Nov 13 20:59:11.015: INFO: Container driver-registrar ready: false, restart count 0 Nov 13 20:59:11.015: INFO: Container mock ready: false, restart count 0 Nov 13 20:59:11.015: INFO: hostexec-k8s-agentpool-24944505-vmss000000 started at 2019-11-13 20:55:51 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:11.015: INFO: Container agnhost ready: true, restart count 0 Nov 13 20:59:11.015: INFO: host-test-container-pod started at 2019-11-13 20:58:55 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:11.015: INFO: Container agnhost ready: false, restart count 0 Nov 13 20:59:11.015: INFO: exec-volume-test-emptydir-kmpp started at 2019-11-13 20:58:55 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:11.015: INFO: Container exec-container-emptydir-kmpp ready: false, restart count 0 Nov 13 20:59:11.015: INFO: csi-hostpathplugin-0 started at 2019-11-13 20:59:07 +0000 UTC (0+0 container statuses recorded) Nov 13 20:59:11.015: INFO: netserver-1 started at 2019-11-13 20:56:58 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:11.015: INFO: Container webserver ready: true, restart count 0 Nov 13 20:59:11.015: INFO: csi-hostpath-attacher-0 started at 2019-11-13 20:58:45 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:11.015: INFO: Container csi-attacher ready: false, restart count 0 Nov 13 20:59:11.015: INFO: test-container-pod started at 2019-11-13 20:58:38 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:11.015: INFO: Container webserver ready: false, restart count 0 Nov 13 20:59:11.015: INFO: keyvault-flexvolume-sn47m started at 2019-11-13 20:42:59 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:11.015: INFO: Container keyvault-flexvolume ready: true, restart count 0 Nov 13 20:59:11.015: INFO: blobfuse-flexvol-installer-drhkj started at 2019-11-13 20:42:59 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:11.015: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 Nov 13 20:59:11.015: INFO: local-injector started at 2019-11-13 20:56:34 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:11.015: INFO: Container local-injector ready: true, restart count 0 Nov 13 20:59:11.015: INFO: httpd started at 2019-11-13 20:55:17 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:11.015: INFO: Container httpd ready: true, restart count 0 Nov 13 20:59:11.015: INFO: hostexec-k8s-agentpool-24944505-vmss000000 started at 2019-11-13 20:58:13 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:11.015: INFO: Container agnhost ready: false, restart count 0 Nov 13 20:59:11.015: INFO: csi-hostpath-provisioner-0 started at 2019-11-13 20:58:50 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:11.015: INFO: Container csi-provisioner ready: false, restart count 0 Nov 13 20:59:11.015: INFO: kube-proxy-h9n4c started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:11.015: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 20:59:11.015: INFO: kubernetes-dashboard-65966766b9-lq4lm started at 2019-11-13 20:42:59 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:11.015: INFO: Container kubernetes-dashboard ready: true, restart count 0 Nov 13 20:59:11.015: INFO: hostexec-k8s-agentpool-24944505-vmss000000 started at 2019-11-13 20:56:40 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:11.015: INFO: Container agnhost ready: true, restart count 0 Nov 13 20:59:11.015: INFO: csi-hostpath-resizer-0 started at 2019-11-13 20:55:57 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:11.015: INFO: Container csi-resizer ready: true, restart count 0 Nov 13 20:59:11.015: INFO: csi-snapshotter-0 started at 2019-11-13 20:55:57 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:11.015: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 13 20:59:11.015: INFO: hostexec-k8s-agentpool-24944505-vmss000000 started at 2019-11-13 20:58:06 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:11.015: INFO: Container agnhost ready: false, restart count 0 Nov 13 20:59:11.015: INFO: failure-4 started at 2019-11-13 20:58:43 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:11.015: INFO: Container failure-4 ready: false, restart count 0 Nov 13 20:59:11.015: INFO: csi-hostpath-resizer-0 started at 2019-11-13 20:58:51 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:11.015: INFO: Container csi-resizer ready: false, restart count 0 Nov 13 20:59:11.015: INFO: hostexec-k8s-agentpool-24944505-vmss000000 started at 2019-11-13 20:59:01 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:11.015: INFO: Container agnhost ready: false, restart count 0 Nov 13 20:59:11.015: INFO: pod-subpath-test-local-preprovisionedpv-t2hj started at 2019-11-13 20:58:41 +0000 UTC (2+2 container statuses recorded) Nov 13 20:59:11.015: INFO: Init container init-volume-local-preprovisionedpv-t2hj ready: false, restart count 0 Nov 13 20:59:11.015: INFO: Init container test-init-subpath-local-preprovisionedpv-t2hj ready: false, restart count 0 Nov 13 20:59:11.015: INFO: Container test-container-subpath-local-preprovisionedpv-t2hj ready: false, restart count 0 Nov 13 20:59:11.015: INFO: Container test-container-volume-local-preprovisionedpv-t2hj ready: false, restart count 0 Nov 13 20:59:11.015: INFO: failure-3-7sgnr started at 2019-11-13 20:57:57 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:11.015: INFO: Container failure-3 ready: true, restart count 0 Nov 13 20:59:11.015: INFO: local-client started at 2019-11-13 20:56:59 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:11.015: INFO: Container local-client ready: true, restart count 0 Nov 13 20:59:11.015: INFO: csi-hostpath-attacher-0 started at 2019-11-13 20:55:56 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:11.015: INFO: Container csi-attacher ready: true, restart count 0 Nov 13 20:59:11.015: INFO: csi-hostpathplugin-0 started at 2019-11-13 20:56:40 +0000 UTC (0+3 container statuses recorded) Nov 13 20:59:11.015: INFO: Container hostpath ready: true, restart count 0 Nov 13 20:59:11.015: INFO: Container liveness-probe ready: true, restart count 0 Nov 13 20:59:11.015: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 13 20:59:11.015: INFO: netserver-0 started at 2019-11-13 20:57:11 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:11.015: INFO: Container webserver ready: true, restart count 0 Nov 13 20:59:11.015: INFO: metrics-server-58ff8c5ddf-r2d7f started at 2019-11-13 20:42:59 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:11.015: INFO: Container metrics-server ready: true, restart count 0 W1113 20:59:11.072029 92631 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 20:59:13.982: INFO: Latency metrics for node k8s-agentpool-24944505-vmss000000 Nov 13 20:59:13.982: INFO: Logging node info for node k8s-agentpool-24944505-vmss000001 Nov 13 20:59:14.038: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool-24944505-vmss000001 /api/v1/nodes/k8s-agentpool-24944505-vmss000001 da675d2e-0aad-4a8f-b94d-1311fb000563 28381 0 2019-11-13 20:42:24 +0000 UTC <nil> <nil> map[agentpool:agentpool beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool-24944505-vmss000001 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: storageprofile:managed storagetier:Premium_LRS] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-351":"k8s-agentpool-24944505-vmss000001","csi-hostpath-ephemeral-5584":"k8s-agentpool-24944505-vmss000001","csi-hostpath-provisioning-1638":"k8s-agentpool-24944505-vmss000001","csi-hostpath-provisioning-225":"k8s-agentpool-24944505-vmss000001","csi-hostpath-provisioning-7099":"k8s-agentpool-24944505-vmss000001","csi-hostpath-volume-6291":"k8s-agentpool-24944505-vmss000001","csi-hostpath-volume-expand-8458":"k8s-agentpool-24944505-vmss000001","csi-mock-csi-mock-volumes-597":"csi-mock-csi-mock-volumes-597","csi-mock-csi-mock-volumes-8812":"csi-mock-csi-mock-volumes-8812","csi-mock-csi-mock-volumes-9088":"csi-mock-csi-mock-volumes-9088"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-24944505-vmss/virtualMachines/1,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},example.com/fakecpu: {{800 0} {<nil>} 800 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16797569024 0} {<nil>} 16403876Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},example.com/fakecpu: {{800 0} {<nil>} 800 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16011137024 0} {<nil>} 15635876Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 20:58:19 +0000 UTC,LastTransitionTime:2019-11-13 20:42:24 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 20:58:19 +0000 UTC,LastTransitionTime:2019-11-13 20:42:24 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 20:58:19 +0000 UTC,LastTransitionTime:2019-11-13 20:42:24 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 20:58:19 +0000 UTC,LastTransitionTime:2019-11-13 20:42:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.248.0.5,},NodeAddress{Type:Hostname,Address:k8s-agentpool-24944505-vmss000001,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:6ed41b2c4b884fccaf94748594531e06,SystemUUID:9CE5C2E3-7C34-C64A-9734-2477E19937B7,BootID:306ac3f6-a23b-423e-83c9-e3d5f3938f6c,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[quay.io/kubernetes_incubator/nfs-provisioner@sha256:df762117e3c891f2d2ddff46ecb0776ba1f9f3c44cfd7739b0683bcd7a7954a8 quay.io/kubernetes_incubator/nfs-provisioner:v2.2.2],SizeBytes:391772778,},ContainerImage{Names:[gcr.io/google-samples/gb-frontend@sha256:35cb427341429fac3df10ff74600ea73e8ec0754d78f9ce89e0b4f3d70d53ba6 gcr.io/google-samples/gb-frontend:v6],SizeBytes:373099368,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/volume/nfs@sha256:c2ad734346f608a5f7d69cfded93c4e8094069320657bd372d12ba21dea3ea71 gcr.io/kubernetes-e2e-test-images/volume/nfs:1.0],SizeBytes:225358913,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/jessie-dnsutils@sha256:ad583e33cb284f7ef046673809b146ec4053cda19b54a85d2b180a86169715eb gcr.io/kubernetes-e2e-test-images/jessie-dnsutils:1.0],SizeBytes:195659796,},ContainerImage{Names:[httpd@sha256:addd70e4ee83f3bc9a4c1c7c41e37927ba47faf639312fc936df3afad7926f5a httpd:2.4.39-alpine],SizeBytes:126894770,},ContainerImage{Names:[httpd@sha256:eb8ccf084cf3e80eece1add239effefd171eb39adbc154d33c14260d905d4060 httpd:2.4.38-alpine],SizeBytes:123781643,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727 gcr.io/kubernetes-e2e-test-images/agnhost:2.6],SizeBytes:57345321,},ContainerImage{Names:[quay.io/k8scsi/csi-provisioner@sha256:0efcb424f1dde9b9fb11a1a14f2e48ab47e1c3f08bc3a929990dcfcb1f7ab34f quay.io/k8scsi/csi-provisioner:v1.4.0-rc1],SizeBytes:54431016,},ContainerImage{Names:[quay.io/k8scsi/csi-snapshotter@sha256:e3d3e742e32d00488fdb401045b9b1d033d7ca0ab6e760f77b24750fc95e5f70 quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1],SizeBytes:51703561,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0],SizeBytes:46226754,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:0aba670b4d9d6b2e720bbf575d733156c676b693ca26501235444490300db838 quay.io/k8scsi/csi-attacher:v1.1.0],SizeBytes:42839085,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:7d46fb6eb8b890dc546029d1565d502b4a1d974d33625c6ee2bc7991b77fc1a1 quay.io/k8scsi/csi-resizer:v0.2.0],SizeBytes:42817100,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:f315c9042e56def3c05c6b04fe79ec9da6d39ddc557ca365a76cf35964ea08b6 quay.io/k8scsi/csi-resizer:v0.1.0],SizeBytes:42623056,},ContainerImage{Names:[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0],SizeBytes:42488424,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nonroot@sha256:d4ede5c74517090b6686219059118ed178cf4620f5db8781b32f806bb1e7395b gcr.io/kubernetes-e2e-test-images/nonroot:1.0],SizeBytes:42321438,},ContainerImage{Names:[redis@sha256:50899ea1ceed33fa03232f3ac57578a424faa1742c1ac9c7a7bdb95cdf19b858 redis:5.0.5-alpine],SizeBytes:29331594,},ContainerImage{Names:[quay.io/k8scsi/hostpathplugin@sha256:b4826e492fc1762fceaf9726f41575ca0a4567864d3d235da874818de18039de quay.io/k8scsi/hostpathplugin:v1.2.0-rc5],SizeBytes:28761497,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7],SizeBytes:22909487,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/echoserver@sha256:e9ba514b896cdf559eef8788b66c2c3ee55f3572df617647b4b0d8b6bf81cf19 gcr.io/kubernetes-e2e-test-images/echoserver:2.2],SizeBytes:21692741,},ContainerImage{Names:[quay.io/k8scsi/mock-driver@sha256:e0eed916b7d970bad2b7d9875f9ad16932f987f0f3d91ec5d86da68b0b5cc9d1 quay.io/k8scsi/mock-driver:v2.1.0],SizeBytes:16226335,},ContainerImage{Names:[nginx@sha256:485b610fefec7ff6c463ced9623314a04ed67e3945b9c08d7e53a47f6d108dc7 nginx:1.14-alpine],SizeBytes:16032814,},ContainerImage{Names:[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0],SizeBytes:15815995,},ContainerImage{Names:[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0],SizeBytes:14967303,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/dnsutils@sha256:2abeee84efb79c14d731966e034af33bf324d3b26ca28497555511ff094b3ddd gcr.io/kubernetes-e2e-test-images/dnsutils:1.1],SizeBytes:9349974,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nautilus@sha256:33a732d4c42a266912a5091598a0f07653c9134db4b8d571690d8afd509e0bfc gcr.io/kubernetes-e2e-test-images/nautilus:1.0],SizeBytes:4753501,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/kitten@sha256:bcbc4875c982ab39aa7c4f6acf4a287f604e996d9f34a3fbda8c3d1a7457d1f6 gcr.io/kubernetes-e2e-test-images/kitten:1.0],SizeBytes:4747037,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/test-webserver@sha256:7f93d6e32798ff28bc6289254d0c2867fe2c849c8e46edc50f8624734309812e gcr.io/kubernetes-e2e-test-images/test-webserver:1.0],SizeBytes:4732240,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 gcr.io/kubernetes-e2e-test-images/mounttest:1.0],SizeBytes:1563521,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea k8s.gcr.io/pause-amd64:3.1 k8s.gcr.io/pause:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 20:59:14.038: INFO: Logging kubelet events for node k8s-agentpool-24944505-vmss000001 Nov 13 20:59:14.102: INFO: Logging pods the kubelet thinks is on node k8s-agentpool-24944505-vmss000001 Nov 13 20:59:14.293: INFO: azure-ip-masq-agent-wwpg7 started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:14.293: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 20:59:14.293: INFO: hostexec-k8s-agentpool-24944505-vmss000001 started at 2019-11-13 20:56:22 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:14.293: INFO: Container agnhost ready: true, restart count 0 Nov 13 20:59:14.293: INFO: rs-pod1-7j6k4 started at 2019-11-13 20:58:27 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:14.293: INFO: Container pod1 ready: false, restart count 0 Nov 13 20:59:14.293: INFO: kube-proxy-7bpf7 started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:14.293: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 20:59:14.293: INFO: failure-1 started at 2019-11-13 20:56:01 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:14.293: INFO: Container failure-1 ready: false, restart count 0 Nov 13 20:59:14.293: INFO: netserver-0 started at 2019-11-13 20:56:58 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:14.293: INFO: Container webserver ready: true, restart count 0 Nov 13 20:59:14.293: INFO: local-injector started at 2019-11-13 20:57:19 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:14.293: INFO: Container local-injector ready: true, restart count 0 Nov 13 20:59:14.293: INFO: rs-pod1-7hl8k started at 2019-11-13 20:58:29 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:14.293: INFO: Container pod1 ready: false, restart count 0 Nov 13 20:59:14.293: INFO: inline-volume-tester-hblrb started at 2019-11-13 20:50:19 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:14.293: INFO: Container csi-volume-tester ready: true, restart count 0 Nov 13 20:59:14.293: INFO: var-expansion-6fcb6f44-0eaf-4ff4-8b7d-88414afa1b73 started at 2019-11-13 20:58:18 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:14.293: INFO: Container dapi-container ready: false, restart count 0 Nov 13 20:59:14.293: INFO: rs-pod1-hzx6b started at 2019-11-13 20:58:25 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:14.293: INFO: Container pod1 ready: false, restart count 0 Nov 13 20:59:14.293: INFO: test-container-pod started at 2019-11-13 20:58:46 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:14.293: INFO: Container webserver ready: false, restart count 0 Nov 13 20:59:14.293: INFO: busybox-readonly-fsf658f934-3681-4d1f-96d9-dc37913b0743 started at 2019-11-13 20:58:43 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:14.293: INFO: Container busybox-readonly-fsf658f934-3681-4d1f-96d9-dc37913b0743 ready: false, restart count 0 Nov 13 20:59:14.293: INFO: send-events-b20e2ffa-1440-482b-a9a5-b7c7fdcfd253 started at 2019-11-13 20:56:42 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:14.293: INFO: Container p ready: true, restart count 0 Nov 13 20:59:14.293: INFO: csi-hostpath-resizer-0 started at 2019-11-13 20:50:19 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:14.293: INFO: Container csi-resizer ready: true, restart count 0 Nov 13 20:59:14.293: INFO: without-label started at 2019-11-13 20:58:22 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:14.293: INFO: Container without-label ready: false, restart count 0 Nov 13 20:59:14.293: INFO: pod-prestop-hook-6a39099a-c80d-43f7-b126-406a4e28d257 started at 2019-11-13 20:56:36 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:14.293: INFO: Container nginx ready: true, restart count 0 Nov 13 20:59:14.293: INFO: host-test-container-pod started at 2019-11-13 20:58:20 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:14.293: INFO: Container agnhost ready: false, restart count 0 Nov 13 20:59:14.293: INFO: failure-2-mgssf started at 2019-11-13 20:56:52 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:14.293: INFO: Container failure-2 ready: true, restart count 1 Nov 13 20:59:14.293: INFO: pod-subpath-test-local-preprovisionedpv-xgnb started at 2019-11-13 20:57:51 +0000 UTC (2+1 container statuses recorded) Nov 13 20:59:14.293: INFO: Init container init-volume-local-preprovisionedpv-xgnb ready: true, restart count 0 Nov 13 20:59:14.293: INFO: Init container test-init-volume-local-preprovisionedpv-xgnb ready: false, restart count 0 Nov 13 20:59:14.293: INFO: Container test-container-subpath-local-preprovisionedpv-xgnb ready: false, restart count 0 Nov 13 20:59:14.293: INFO: image-pull-testfe67cfa6-713e-48b7-a507-0fa7c09ba3c9 started at 2019-11-13 20:58:40 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:14.293: INFO: Container image-pull-test ready: false, restart count 0 Nov 13 20:59:14.293: INFO: csi-hostpathplugin-0 started at 2019-11-13 20:55:41 +0000 UTC (0+3 container statuses recorded) Nov 13 20:59:14.293: INFO: Container hostpath ready: true, restart count 0 Nov 13 20:59:14.293: INFO: Container liveness-probe ready: true, restart count 0 Nov 13 20:59:14.293: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 13 20:59:14.293: INFO: success started at 2019-11-13 20:55:42 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:14.293: INFO: Container success ready: false, restart count 0 Nov 13 20:59:14.293: INFO: test-container-pod started at 2019-11-13 20:58:19 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:14.293: INFO: Container webserver ready: false, restart count 0 Nov 13 20:59:14.293: INFO: netserver-1 started at 2019-11-13 20:57:54 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:14.293: INFO: Container webserver ready: true, restart count 0 Nov 13 20:59:14.293: INFO: hostexec-k8s-agentpool-24944505-vmss000001 started at 2019-11-13 20:58:39 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:14.293: INFO: Container agnhost ready: false, restart count 0 Nov 13 20:59:14.293: INFO: coredns-87f5d796-jd8bc started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:14.293: INFO: Container coredns ready: true, restart count 0 Nov 13 20:59:14.293: INFO: csi-snapshotter-0 started at 2019-11-13 20:50:19 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:14.293: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 13 20:59:14.293: INFO: hostexec-k8s-agentpool-24944505-vmss000001 started at 2019-11-13 20:56:17 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:14.293: INFO: Container agnhost ready: true, restart count 0 Nov 13 20:59:14.293: INFO: netserver-1 started at 2019-11-13 20:57:12 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:14.293: INFO: Container webserver ready: true, restart count 0 Nov 13 20:59:14.293: INFO: host-test-container-pod started at 2019-11-13 20:58:38 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:14.293: INFO: Container agnhost ready: false, restart count 0 Nov 13 20:59:14.293: INFO: keyvault-flexvolume-7phpr started at 2019-11-13 20:42:58 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:14.293: INFO: Container keyvault-flexvolume ready: true, restart count 0 Nov 13 20:59:14.293: INFO: csi-hostpath-provisioner-0 started at 2019-11-13 20:50:19 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:14.293: INFO: Container csi-provisioner ready: true, restart count 0 Nov 13 20:59:14.293: INFO: rs-pod1-cnmm6 started at 2019-11-13 20:58:24 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:14.293: INFO: Container pod1 ready: false, restart count 0 Nov 13 20:59:14.293: INFO: sample-webhook-deployment-86d95b659d-zf99v started at 2019-11-13 20:58:42 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:14.293: INFO: Container sample-webhook ready: false, restart count 0 Nov 13 20:59:14.293: INFO: blobfuse-flexvol-installer-cpm6h started at 2019-11-13 20:42:58 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:14.293: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 Nov 13 20:59:14.293: INFO: pfpod started at 2019-11-13 20:58:16 +0000 UTC (0+2 container statuses recorded) Nov 13 20:59:14.293: INFO: Container portforwardtester ready: false, restart count 0 Nov 13 20:59:14.293: INFO: Container readiness ready: false, restart count 0 Nov 13 20:59:14.293: INFO: rs-pod1-xsqgc started at 2019-11-13 20:58:27 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:14.293: INFO: Container pod1 ready: false, restart count 0 Nov 13 20:59:14.293: INFO: csi-hostpath-attacher-0 started at 2019-11-13 20:50:18 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:14.293: INFO: Container csi-attacher ready: true, restart count 0 Nov 13 20:59:14.293: INFO: pod-subpath-test-emptydir-w7dj started at 2019-11-13 20:58:37 +0000 UTC (1+2 container statuses recorded) Nov 13 20:59:14.293: INFO: Init container test-init-subpath-emptydir-w7dj ready: false, restart count 0 Nov 13 20:59:14.293: INFO: Container test-container-subpath-emptydir-w7dj ready: false, restart count 0 Nov 13 20:59:14.293: INFO: Container test-container-volume-emptydir-w7dj ready: false, restart count 0 W1113 20:59:14.350943 92631 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 20:59:14.896: INFO: Latency metrics for node k8s-agentpool-24944505-vmss000001 Nov 13 20:59:14.896: INFO: Logging node info for node k8s-master-24944505-vmss000000 Nov 13 20:59:14.952: INFO: Node Info: &Node{ObjectMeta:{k8s-master-24944505-vmss000000 /api/v1/nodes/k8s-master-24944505-vmss000000 048fbae2-b97f-43a2-b471-398180136f12 29509 0 2019-11-13 20:42:13 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-24944505-vmss000000 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-24944505-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284887552 0} {<nil>} 7114148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6498455552 0} {<nil>} 6346148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 20:59:06 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 20:59:06 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 20:59:06 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 20:59:06 +0000 UTC,LastTransitionTime:2019-11-13 20:42:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.4,},NodeAddress{Type:Hostname,Address:k8s-master-24944505-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:7bd754c0dc8647518c3043bcae3a76cc,SystemUUID:FA5CFE99-8148-6C46-B4E6-5EA2A5DE9D2A,BootID:74f7b0a2-e85c-438e-a642-c5b32900bb28,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:3237a53b135d8692ade82eca845fd0259ecbfcbe05fad35400e7d5c2018c4abd k8sprow.azurecr.io/azure-cloud-controller-manager:1194710255657816064-e688ecb0],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 20:59:14.952: INFO: Logging kubelet events for node k8s-master-24944505-vmss000000 Nov 13 20:59:15.012: INFO: Logging pods the kubelet thinks is on node k8s-master-24944505-vmss000000 Nov 13 20:59:15.089: INFO: cloud-controller-manager-k8s-master-24944505-vmss000000 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:15.089: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 13 20:59:15.089: INFO: kube-addon-manager-k8s-master-24944505-vmss000000 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:15.089: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 13 20:59:15.089: INFO: kube-apiserver-k8s-master-24944505-vmss000000 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:15.089: INFO: Container kube-apiserver ready: true, restart count 0 Nov 13 20:59:15.089: INFO: kube-controller-manager-k8s-master-24944505-vmss000000 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:15.089: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 13 20:59:15.089: INFO: kube-scheduler-k8s-master-24944505-vmss000000 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:15.089: INFO: Container kube-scheduler ready: true, restart count 0 Nov 13 20:59:15.089: INFO: azure-ip-masq-agent-6pnst started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:15.089: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 20:59:15.089: INFO: kube-proxy-d9jjh started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:15.089: INFO: Container kube-proxy ready: true, restart count 0 W1113 20:59:15.151187 92631 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 20:59:15.277: INFO: Latency metrics for node k8s-master-24944505-vmss000000 Nov 13 20:59:15.277: INFO: Logging node info for node k8s-master-24944505-vmss000001 Nov 13 20:59:15.332: INFO: Node Info: &Node{ObjectMeta:{k8s-master-24944505-vmss000001 /api/v1/nodes/k8s-master-24944505-vmss000001 a2619878-b119-44f9-b89f-4073bebbb56c 29478 0 2019-11-13 20:42:40 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-24944505-vmss000001 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.6.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-24944505-vmss/virtualMachines/1,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.6.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284883456 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6498451456 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:43:08 +0000 UTC,LastTransitionTime:2019-11-13 20:43:08 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 20:59:02 +0000 UTC,LastTransitionTime:2019-11-13 20:42:34 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 20:59:02 +0000 UTC,LastTransitionTime:2019-11-13 20:42:34 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 20:59:02 +0000 UTC,LastTransitionTime:2019-11-13 20:42:34 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 20:59:02 +0000 UTC,LastTransitionTime:2019-11-13 20:42:40 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.5,},NodeAddress{Type:Hostname,Address:k8s-master-24944505-vmss000001,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:c866a7c1ed64481990416e24387dd317,SystemUUID:EBE64557-2AC8-5740-9FFE-AB9F12412FBC,BootID:3dbb0e0e-0479-4821-bc3f-a8269d250e4c,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:3237a53b135d8692ade82eca845fd0259ecbfcbe05fad35400e7d5c2018c4abd k8sprow.azurecr.io/azure-cloud-controller-manager:1194710255657816064-e688ecb0],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 20:59:15.333: INFO: Logging kubelet events for node k8s-master-24944505-vmss000001 Nov 13 20:59:15.391: INFO: Logging pods the kubelet thinks is on node k8s-master-24944505-vmss000001 Nov 13 20:59:15.472: INFO: kube-controller-manager-k8s-master-24944505-vmss000001 started at 2019-11-13 20:42:28 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:15.472: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 13 20:59:15.472: INFO: kube-proxy-w4bz5 started at 2019-11-13 20:42:41 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:15.472: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 20:59:15.472: INFO: azure-ip-masq-agent-bs52d started at 2019-11-13 20:42:41 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:15.472: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 20:59:15.472: INFO: kube-scheduler-k8s-master-24944505-vmss000001 started at 2019-11-13 20:42:28 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:15.472: INFO: Container kube-scheduler ready: true, restart count 0 Nov 13 20:59:15.472: INFO: cloud-controller-manager-k8s-master-24944505-vmss000001 started at 2019-11-13 20:42:28 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:15.472: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 13 20:59:15.472: INFO: kube-addon-manager-k8s-master-24944505-vmss000001 started at 2019-11-13 20:42:28 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:15.472: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 13 20:59:15.472: INFO: kube-apiserver-k8s-master-24944505-vmss000001 started at 2019-11-13 20:42:28 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:15.472: INFO: Container kube-apiserver ready: true, restart count 0 W1113 20:59:15.528678 92631 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 20:59:15.682: INFO: Latency metrics for node k8s-master-24944505-vmss000001 Nov 13 20:59:15.683: INFO: Logging node info for node k8s-master-24944505-vmss000002 Nov 13 20:59:15.740: INFO: Node Info: &Node{ObjectMeta:{k8s-master-24944505-vmss000002 /api/v1/nodes/k8s-master-24944505-vmss000002 f3b05267-a009-4dba-9f97-4f772e48707e 29508 0 2019-11-13 20:42:13 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-24944505-vmss000002 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.4.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-24944505-vmss/virtualMachines/2,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284887552 0} {<nil>} 7114148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6498455552 0} {<nil>} 6346148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 20:59:06 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 20:59:06 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 20:59:06 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 20:59:06 +0000 UTC,LastTransitionTime:2019-11-13 20:42:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.6,},NodeAddress{Type:Hostname,Address:k8s-master-24944505-vmss000002,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:fc1777f37ced4b9c97acb9b30e276c4f,SystemUUID:E9DF19DC-D02C-714C-A7D6-0BA6CE959C55,BootID:622958a8-d45d-4edf-9266-4dbb759e2359,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:3237a53b135d8692ade82eca845fd0259ecbfcbe05fad35400e7d5c2018c4abd k8sprow.azurecr.io/azure-cloud-controller-manager:1194710255657816064-e688ecb0],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 20:59:15.741: INFO: Logging kubelet events for node k8s-master-24944505-vmss000002 Nov 13 20:59:15.799: INFO: Logging pods the kubelet thinks is on node k8s-master-24944505-vmss000002 Nov 13 20:59:15.876: INFO: kube-apiserver-k8s-master-24944505-vmss000002 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:15.876: INFO: Container kube-apiserver ready: true, restart count 0 Nov 13 20:59:15.876: INFO: kube-controller-manager-k8s-master-24944505-vmss000002 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:15.876: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 13 20:59:15.876: INFO: kube-scheduler-k8s-master-24944505-vmss000002 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:15.876: INFO: Container kube-scheduler ready: true, restart count 0 Nov 13 20:59:15.876: INFO: cloud-controller-manager-k8s-master-24944505-vmss000002 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:15.876: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 13 20:59:15.876: INFO: azure-ip-masq-agent-6skf2 started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:15.876: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 20:59:15.876: INFO: kube-proxy-xwz78 started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:15.876: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 20:59:15.876: INFO: kube-addon-manager-k8s-master-24944505-vmss000002 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:15.876: INFO: Container kube-addon-manager ready: true, restart count 0 W1113 20:59:15.935419 92631 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 20:59:16.087: INFO: Latency metrics for node k8s-master-24944505-vmss000002 Nov 13 20:59:16.087: INFO: Logging node info for node k8s-master-24944505-vmss000003 Nov 13 20:59:16.145: INFO: Node Info: &Node{ObjectMeta:{k8s-master-24944505-vmss000003 /api/v1/nodes/k8s-master-24944505-vmss000003 888cc19e-c62d-4945-a425-97d14e38a0b8 29462 0 2019-11-13 20:42:35 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-24944505-vmss000003 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.5.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-24944505-vmss/virtualMachines/3,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.5.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284887552 0} {<nil>} 7114148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6498455552 0} {<nil>} 6346148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 20:58:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:29 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 20:58:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:29 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 20:58:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:29 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 20:58:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:35 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.7,},NodeAddress{Type:Hostname,Address:k8s-master-24944505-vmss000003,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:2da5fe5a442b4e178a2b62ba91366e4b,SystemUUID:D2D1E816-D70E-3D40-B959-B00A2B76A83A,BootID:c5929aa1-3dac-45d5-bb2c-174c73c7300e,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:3237a53b135d8692ade82eca845fd0259ecbfcbe05fad35400e7d5c2018c4abd k8sprow.azurecr.io/azure-cloud-controller-manager:1194710255657816064-e688ecb0],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 20:59:16.145: INFO: Logging kubelet events for node k8s-master-24944505-vmss000003 Nov 13 20:59:16.204: INFO: Logging pods the kubelet thinks is on node k8s-master-24944505-vmss000003 Nov 13 20:59:16.283: INFO: kube-controller-manager-k8s-master-24944505-vmss000003 started at 2019-11-13 20:42:23 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:16.283: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 13 20:59:16.283: INFO: kube-scheduler-k8s-master-24944505-vmss000003 started at 2019-11-13 20:42:23 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:16.283: INFO: Container kube-scheduler ready: true, restart count 0 Nov 13 20:59:16.283: INFO: cloud-controller-manager-k8s-master-24944505-vmss000003 started at 2019-11-13 20:42:44 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:16.283: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 13 20:59:16.283: INFO: azure-ip-masq-agent-nlqss started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:16.283: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 20:59:16.283: INFO: kube-proxy-kgh5h started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:16.283: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 20:59:16.283: INFO: kube-addon-manager-k8s-master-24944505-vmss000003 started at 2019-11-13 20:42:22 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:16.283: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 13 20:59:16.283: INFO: kube-apiserver-k8s-master-24944505-vmss000003 started at 2019-11-13 20:42:23 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:16.283: INFO: Container kube-apiserver ready: true, restart count 0 W1113 20:59:16.339846 92631 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 20:59:16.469: INFO: Latency metrics for node k8s-master-24944505-vmss000003 Nov 13 20:59:16.469: INFO: Logging node info for node k8s-master-24944505-vmss000004 Nov 13 20:59:16.524: INFO: Node Info: &Node{ObjectMeta:{k8s-master-24944505-vmss000004 /api/v1/nodes/k8s-master-24944505-vmss000004 a26d844f-0612-406b-9a65-7e2ac952a2d5 29511 0 2019-11-13 20:42:14 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-24944505-vmss000004 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-24944505-vmss/virtualMachines/4,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284887552 0} {<nil>} 7114148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6498455552 0} {<nil>} 6346148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 20:59:06 +0000 UTC,LastTransitionTime:2019-11-13 20:42:09 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 20:59:06 +0000 UTC,LastTransitionTime:2019-11-13 20:42:09 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 20:59:06 +0000 UTC,LastTransitionTime:2019-11-13 20:42:09 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 20:59:06 +0000 UTC,LastTransitionTime:2019-11-13 20:42:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.8,},NodeAddress{Type:Hostname,Address:k8s-master-24944505-vmss000004,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:2208959aca1045d9b48fce8d784c1e41,SystemUUID:E253DB3A-3A4D-5648-8169-B2E2976CFB4B,BootID:09bc6101-2da9-4c8e-8274-43db8c5fe1cf,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:3237a53b135d8692ade82eca845fd0259ecbfcbe05fad35400e7d5c2018c4abd k8sprow.azurecr.io/azure-cloud-controller-manager:1194710255657816064-e688ecb0],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 20:59:16.524: INFO: Logging kubelet events for node k8s-master-24944505-vmss000004 Nov 13 20:59:16.583: INFO: Logging pods the kubelet thinks is on node k8s-master-24944505-vmss000004 Nov 13 20:59:16.664: INFO: kube-apiserver-k8s-master-24944505-vmss000004 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:16.664: INFO: Container kube-apiserver ready: true, restart count 0 Nov 13 20:59:16.664: INFO: kube-controller-manager-k8s-master-24944505-vmss000004 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:16.664: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 13 20:59:16.664: INFO: kube-scheduler-k8s-master-24944505-vmss000004 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:16.664: INFO: Container kube-scheduler ready: true, restart count 0 Nov 13 20:59:16.664: INFO: azure-ip-masq-agent-xnw2h started at 2019-11-13 20:42:38 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:16.664: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 20:59:16.664: INFO: kube-proxy-fld6p started at 2019-11-13 20:42:38 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:16.664: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 20:59:16.664: INFO: cloud-controller-manager-k8s-master-24944505-vmss000004 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:16.664: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 13 20:59:16.664: INFO: kube-addon-manager-k8s-master-24944505-vmss000004 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 20:59:16.664: INFO: Container kube-addon-manager ready: true, restart count 0 W1113 20:59:16.719929 92631 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 20:59:16.850: INFO: Latency metrics for node k8s-master-24944505-vmss000004 Nov 13 20:59:16.850: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "sched-preemption-path-5324" for this suite. Nov 13 21:00:09.083: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered Nov 13 21:00:10.944: INFO: namespace sched-preemption-path-5324 deletion completed in 54.036659249s [AfterEach] [sig-scheduling] PreemptionExecutionPath test/e2e/scheduling/preemption.go:274 Nov 13 21:00:10.999: INFO: List existing priorities: Nov 13 21:00:10.999: INFO: p1/1 created at 2019-11-13 20:58:09 +0000 UTC Nov 13 21:00:10.999: INFO: p2/2 created at 2019-11-13 20:58:10 +0000 UTC Nov 13 21:00:10.999: INFO: p3/3 created at 2019-11-13 20:58:10 +0000 UTC Nov 13 21:00:10.999: INFO: p4/4 created at 2019-11-13 20:58:10 +0000 UTC Nov 13 21:00:10.999: INFO: system-cluster-critical/2000000000 created at 2019-11-13 20:42:14 +0000 UTC Nov 13 21:00:10.999: INFO: system-node-critical/2000001000 created at 2019-11-13 20:42:14 +0000 UTC
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[sig\-storage\]\sCSI\sVolumes\s\[Driver\:\scsi\-hostpath\]\s\[Testpattern\:\sDynamic\sPV\s\(default\sfs\)\(allowExpansion\)\]\svolume\-expand\sVerify\sif\soffline\sPVC\sexpansion\sworks$'
test/e2e/storage/testsuites/volume_expand.go:154 Nov 13 21:05:05.409: Unexpected error: <*errors.errorString | 0xc002304c70>: { s: "PersistentVolumeClaims [csi-hostpathvlpd6] not all in phase Bound within 5m0s", } PersistentVolumeClaims [csi-hostpathvlpd6] not all in phase Bound within 5m0s occurred test/e2e/storage/testsuites/base.go:366from junit_26.xml
[BeforeEach] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand test/e2e/storage/testsuites/base.go:93 [BeforeEach] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand test/e2e/storage/testsuites/volume_expand.go:85 [BeforeEach] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand test/e2e/framework/framework.go:151 �[1mSTEP�[0m: Creating a kubernetes client Nov 13 21:00:02.115: INFO: >>> kubeConfig: /workspace/aks880331199/kubeconfig/kubeconfig.westus2.json �[1mSTEP�[0m: Building a namespace api object, basename volume-expand �[1mSTEP�[0m: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in volume-expand-434 �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace [It] Verify if offline PVC expansion works test/e2e/storage/testsuites/volume_expand.go:154 �[1mSTEP�[0m: deploying csi-hostpath driver Nov 13 21:00:02.697: INFO: creating *v1.ServiceAccount: volume-expand-434/csi-attacher Nov 13 21:00:02.758: INFO: creating *v1.ClusterRole: external-attacher-runner-volume-expand-434 Nov 13 21:00:02.758: INFO: Define cluster role external-attacher-runner-volume-expand-434 Nov 13 21:00:02.817: INFO: creating *v1.ClusterRoleBinding: csi-attacher-role-volume-expand-434 Nov 13 21:00:02.880: INFO: creating *v1.Role: volume-expand-434/external-attacher-cfg-volume-expand-434 Nov 13 21:00:02.940: INFO: creating *v1.RoleBinding: volume-expand-434/csi-attacher-role-cfg Nov 13 21:00:03.000: INFO: creating *v1.ServiceAccount: volume-expand-434/csi-provisioner Nov 13 21:00:03.065: INFO: creating *v1.ClusterRole: external-provisioner-runner-volume-expand-434 Nov 13 21:00:03.066: INFO: Define cluster role external-provisioner-runner-volume-expand-434 Nov 13 21:00:03.124: INFO: creating *v1.ClusterRoleBinding: csi-provisioner-role-volume-expand-434 Nov 13 21:00:03.184: INFO: creating *v1.Role: volume-expand-434/external-provisioner-cfg-volume-expand-434 Nov 13 21:00:03.242: INFO: creating *v1.RoleBinding: volume-expand-434/csi-provisioner-role-cfg Nov 13 21:00:03.302: INFO: creating *v1.ServiceAccount: volume-expand-434/csi-snapshotter Nov 13 21:00:03.366: INFO: creating *v1.ClusterRole: external-snapshotter-runner-volume-expand-434 Nov 13 21:00:03.366: INFO: Define cluster role external-snapshotter-runner-volume-expand-434 Nov 13 21:00:03.435: INFO: creating *v1.ClusterRoleBinding: csi-snapshotter-role-volume-expand-434 Nov 13 21:00:03.494: INFO: creating *v1.Role: volume-expand-434/external-snapshotter-leaderelection-volume-expand-434 Nov 13 21:00:03.560: INFO: creating *v1.RoleBinding: volume-expand-434/external-snapshotter-leaderelection Nov 13 21:00:03.627: INFO: creating *v1.ServiceAccount: volume-expand-434/csi-resizer Nov 13 21:00:03.693: INFO: creating *v1.ClusterRole: external-resizer-runner-volume-expand-434 Nov 13 21:00:03.693: INFO: Define cluster role external-resizer-runner-volume-expand-434 Nov 13 21:00:03.753: INFO: creating *v1.ClusterRoleBinding: csi-resizer-role-volume-expand-434 Nov 13 21:00:03.811: INFO: creating *v1.Role: volume-expand-434/external-resizer-cfg-volume-expand-434 Nov 13 21:00:03.875: INFO: creating *v1.RoleBinding: volume-expand-434/csi-resizer-role-cfg Nov 13 21:00:03.934: INFO: creating *v1.Service: volume-expand-434/csi-hostpath-attacher Nov 13 21:00:04.017: INFO: creating *v1.StatefulSet: volume-expand-434/csi-hostpath-attacher Nov 13 21:00:04.082: INFO: creating *v1beta1.CSIDriver: csi-hostpath-volume-expand-434 Nov 13 21:00:04.146: INFO: creating *v1.Service: volume-expand-434/csi-hostpathplugin Nov 13 21:00:04.222: INFO: creating *v1.StatefulSet: volume-expand-434/csi-hostpathplugin Nov 13 21:00:04.287: INFO: creating *v1.Service: volume-expand-434/csi-hostpath-provisioner Nov 13 21:00:04.361: INFO: creating *v1.StatefulSet: volume-expand-434/csi-hostpath-provisioner Nov 13 21:00:04.422: INFO: creating *v1.Service: volume-expand-434/csi-hostpath-resizer Nov 13 21:00:04.510: INFO: creating *v1.StatefulSet: volume-expand-434/csi-hostpath-resizer Nov 13 21:00:04.586: INFO: creating *v1.Service: volume-expand-434/csi-snapshotter Nov 13 21:00:04.661: INFO: creating *v1.StatefulSet: volume-expand-434/csi-snapshotter Nov 13 21:00:04.727: INFO: creating *v1.ClusterRoleBinding: psp-csi-hostpath-role-volume-expand-434 Nov 13 21:00:04.794: INFO: Test running for native CSI Driver, not checking metrics Nov 13 21:00:04.794: INFO: Creating resource for dynamic PV �[1mSTEP�[0m: creating a StorageClass volume-expand-434-csi-hostpath-volume-expand-434-sc2lhhr �[1mSTEP�[0m: creating a claim Nov 13 21:00:04.854: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil Nov 13 21:00:04.921: INFO: Waiting up to 5m0s for PersistentVolumeClaims [csi-hostpathvlpd6] to have phase Bound Nov 13 21:00:04.976: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:00:07.033: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:00:09.090: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:00:11.146: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:00:13.211: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:00:15.267: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:00:17.323: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:00:19.381: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:00:21.438: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:00:23.494: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:00:25.584: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:00:27.641: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:00:29.699: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:00:31.755: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:00:33.811: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:00:35.868: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:00:37.923: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:00:39.980: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:00:42.036: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:00:44.091: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:00:46.148: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:00:48.204: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:00:50.261: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:00:52.321: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:00:54.378: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:00:56.436: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:00:58.493: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:01:00.549: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:01:02.607: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:01:04.663: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:01:06.724: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:01:08.780: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:01:10.835: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:01:12.892: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:01:14.948: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:01:17.007: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:01:19.073: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:01:21.129: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:01:23.185: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:01:25.242: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:01:27.301: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:01:29.361: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:01:31.417: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:01:33.472: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:01:35.528: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:01:37.584: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:01:39.656: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:01:41.712: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:01:43.769: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:01:45.825: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:01:47.881: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:01:49.938: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:01:51.997: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:01:54.053: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:01:56.109: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:01:58.168: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:02:00.227: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:02:02.282: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:02:04.340: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:02:06.396: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:02:08.452: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:02:10.511: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:02:12.567: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:02:14.625: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:02:16.681: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:02:18.736: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:02:20.794: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:02:22.851: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:02:24.906: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:02:26.962: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:02:29.019: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:02:31.076: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:02:33.133: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:02:35.188: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:02:37.244: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:02:39.300: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:02:41.378: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:02:43.434: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:02:45.491: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:02:47.546: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:02:49.637: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:02:51.694: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:02:53.751: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:02:55.808: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:02:57.864: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:02:59.921: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:03:01.984: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:03:04.040: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:03:06.098: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:03:08.156: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:03:10.213: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:03:12.269: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:03:14.325: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:03:16.380: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:03:18.438: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:03:20.494: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:03:22.564: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:03:24.621: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:03:26.679: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:03:28.739: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:03:30.796: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:03:32.851: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:03:34.908: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:03:36.964: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:03:39.020: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:03:41.076: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:03:43.140: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:03:45.203: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:03:47.258: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:03:49.314: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:03:51.371: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:03:53.429: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:03:55.486: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:03:57.546: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:03:59.608: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:04:01.665: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:04:03.726: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:04:05.782: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:04:07.839: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:04:09.895: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:04:11.955: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:04:14.011: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:04:16.068: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:04:18.124: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:04:20.180: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:04:22.236: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:04:24.291: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:04:26.348: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:04:28.407: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:04:30.463: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:04:32.527: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:04:34.586: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:04:36.642: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:04:38.701: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:04:40.761: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:04:42.825: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:04:44.886: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:04:46.954: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:04:49.010: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:04:51.066: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:04:53.127: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:04:55.184: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:04:57.239: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:04:59.296: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:05:01.352: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:05:03.409: INFO: PersistentVolumeClaim csi-hostpathvlpd6 found but phase is Pending instead of Bound. Nov 13 21:05:05.409: FAIL: Unexpected error: <*errors.errorString | 0xc002304c70>: { s: "PersistentVolumeClaims [csi-hostpathvlpd6] not all in phase Bound within 5m0s", } PersistentVolumeClaims [csi-hostpathvlpd6] not all in phase Bound within 5m0s occurred [AfterEach] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand test/e2e/framework/framework.go:152 �[1mSTEP�[0m: Collecting events from namespace "volume-expand-434". �[1mSTEP�[0m: Found 91 events. Nov 13 21:05:05.570: INFO: At 2019-11-13 21:00:04 +0000 UTC - event for csi-hostpath-attacher: {statefulset-controller } SuccessfulCreate: create Pod csi-hostpath-attacher-0 in StatefulSet csi-hostpath-attacher successful Nov 13 21:05:05.570: INFO: At 2019-11-13 21:00:04 +0000 UTC - event for csi-hostpath-provisioner: {statefulset-controller } SuccessfulCreate: create Pod csi-hostpath-provisioner-0 in StatefulSet csi-hostpath-provisioner successful Nov 13 21:05:05.570: INFO: At 2019-11-13 21:00:04 +0000 UTC - event for csi-hostpath-resizer: {statefulset-controller } SuccessfulCreate: create Pod csi-hostpath-resizer-0 in StatefulSet csi-hostpath-resizer successful Nov 13 21:05:05.570: INFO: At 2019-11-13 21:00:04 +0000 UTC - event for csi-hostpathplugin: {statefulset-controller } SuccessfulCreate: create Pod csi-hostpathplugin-0 in StatefulSet csi-hostpathplugin successful Nov 13 21:05:05.570: INFO: At 2019-11-13 21:00:04 +0000 UTC - event for csi-hostpathvlpd6: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "csi-hostpath-volume-expand-434" or manually created by system administrator Nov 13 21:05:05.570: INFO: At 2019-11-13 21:00:04 +0000 UTC - event for csi-snapshotter: {statefulset-controller } SuccessfulCreate: create Pod csi-snapshotter-0 in StatefulSet csi-snapshotter successful Nov 13 21:05:05.570: INFO: At 2019-11-13 21:00:10 +0000 UTC - event for csi-hostpathplugin: {statefulset-controller } SuccessfulDelete: delete Pod csi-hostpathplugin-0 in StatefulSet csi-hostpathplugin successful Nov 13 21:05:05.570: INFO: At 2019-11-13 21:00:10 +0000 UTC - event for csi-hostpathplugin: {statefulset-controller } RecreatingFailedPod: StatefulSet volume-expand-434/csi-hostpathplugin is recreating failed Pod csi-hostpathplugin-0 Nov 13 21:05:05.570: INFO: At 2019-11-13 21:00:10 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.570: INFO: At 2019-11-13 21:00:20 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.570: INFO: At 2019-11-13 21:00:29 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.570: INFO: At 2019-11-13 21:00:34 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.570: INFO: At 2019-11-13 21:00:37 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.570: INFO: At 2019-11-13 21:00:41 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.570: INFO: At 2019-11-13 21:00:43 +0000 UTC - event for csi-hostpath-attacher-0: {kubelet k8s-agentpool-24944505-vmss000000} Pulling: Pulling image "quay.io/k8scsi/csi-attacher:v1.2.0" Nov 13 21:05:05.570: INFO: At 2019-11-13 21:00:44 +0000 UTC - event for csi-hostpath-provisioner-0: {kubelet k8s-agentpool-24944505-vmss000000} Pulling: Pulling image "quay.io/k8scsi/csi-provisioner:v1.4.0-rc1" Nov 13 21:05:05.570: INFO: At 2019-11-13 21:00:47 +0000 UTC - event for csi-hostpath-attacher-0: {kubelet k8s-agentpool-24944505-vmss000000} Pulled: Successfully pulled image "quay.io/k8scsi/csi-attacher:v1.2.0" Nov 13 21:05:05.570: INFO: At 2019-11-13 21:00:47 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.570: INFO: At 2019-11-13 21:00:51 +0000 UTC - event for csi-hostpath-provisioner-0: {kubelet k8s-agentpool-24944505-vmss000000} Pulled: Successfully pulled image "quay.io/k8scsi/csi-provisioner:v1.4.0-rc1" Nov 13 21:05:05.570: INFO: At 2019-11-13 21:00:52 +0000 UTC - event for csi-hostpath-attacher-0: {kubelet k8s-agentpool-24944505-vmss000000} Created: Created container csi-attacher Nov 13 21:05:05.570: INFO: At 2019-11-13 21:00:52 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.570: INFO: At 2019-11-13 21:00:56 +0000 UTC - event for csi-snapshotter-0: {kubelet k8s-agentpool-24944505-vmss000000} Pulling: Pulling image "quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1" Nov 13 21:05:05.570: INFO: At 2019-11-13 21:00:57 +0000 UTC - event for csi-hostpath-provisioner-0: {kubelet k8s-agentpool-24944505-vmss000000} Created: Created container csi-provisioner Nov 13 21:05:05.570: INFO: At 2019-11-13 21:00:57 +0000 UTC - event for csi-hostpath-resizer-0: {kubelet k8s-agentpool-24944505-vmss000000} Pulling: Pulling image "quay.io/k8scsi/csi-resizer:v0.2.0" Nov 13 21:05:05.570: INFO: At 2019-11-13 21:00:57 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.570: INFO: At 2019-11-13 21:00:59 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.570: INFO: At 2019-11-13 21:00:59 +0000 UTC - event for csi-snapshotter-0: {kubelet k8s-agentpool-24944505-vmss000000} Pulled: Successfully pulled image "quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1" Nov 13 21:05:05.570: INFO: At 2019-11-13 21:01:00 +0000 UTC - event for csi-hostpath-attacher-0: {kubelet k8s-agentpool-24944505-vmss000000} Started: Started container csi-attacher Nov 13 21:05:05.570: INFO: At 2019-11-13 21:01:02 +0000 UTC - event for csi-hostpath-provisioner-0: {kubelet k8s-agentpool-24944505-vmss000000} Started: Started container csi-provisioner Nov 13 21:05:05.570: INFO: At 2019-11-13 21:01:02 +0000 UTC - event for csi-hostpath-resizer-0: {kubelet k8s-agentpool-24944505-vmss000000} Pulled: Successfully pulled image "quay.io/k8scsi/csi-resizer:v0.2.0" Nov 13 21:05:05.570: INFO: At 2019-11-13 21:01:03 +0000 UTC - event for csi-snapshotter-0: {kubelet k8s-agentpool-24944505-vmss000000} Created: Created container csi-snapshotter Nov 13 21:05:05.570: INFO: At 2019-11-13 21:01:05 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.570: INFO: At 2019-11-13 21:01:08 +0000 UTC - event for csi-hostpath-resizer-0: {kubelet k8s-agentpool-24944505-vmss000000} Created: Created container csi-resizer Nov 13 21:05:05.570: INFO: At 2019-11-13 21:01:10 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.570: INFO: At 2019-11-13 21:01:10 +0000 UTC - event for csi-snapshotter-0: {kubelet k8s-agentpool-24944505-vmss000000} Started: Started container csi-snapshotter Nov 13 21:05:05.570: INFO: At 2019-11-13 21:01:14 +0000 UTC - event for csi-hostpath-resizer-0: {kubelet k8s-agentpool-24944505-vmss000000} Started: Started container csi-resizer Nov 13 21:05:05.570: INFO: At 2019-11-13 21:01:18 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.570: INFO: At 2019-11-13 21:01:24 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.570: INFO: At 2019-11-13 21:01:30 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.570: INFO: At 2019-11-13 21:01:35 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.570: INFO: At 2019-11-13 21:01:38 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.570: INFO: At 2019-11-13 21:01:41 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.570: INFO: At 2019-11-13 21:01:46 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.570: INFO: At 2019-11-13 21:01:49 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.570: INFO: At 2019-11-13 21:01:54 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.570: INFO: At 2019-11-13 21:01:59 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.570: INFO: At 2019-11-13 21:02:03 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.570: INFO: At 2019-11-13 21:02:07 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.570: INFO: At 2019-11-13 21:02:15 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.570: INFO: At 2019-11-13 21:02:23 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.570: INFO: At 2019-11-13 21:02:24 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.570: INFO: At 2019-11-13 21:02:27 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.570: INFO: At 2019-11-13 21:02:29 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.571: INFO: At 2019-11-13 21:02:31 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.571: INFO: At 2019-11-13 21:02:33 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.571: INFO: At 2019-11-13 21:02:38 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.571: INFO: At 2019-11-13 21:02:52 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.571: INFO: At 2019-11-13 21:02:57 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.571: INFO: At 2019-11-13 21:03:02 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.571: INFO: At 2019-11-13 21:03:03 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.571: INFO: At 2019-11-13 21:03:04 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.571: INFO: At 2019-11-13 21:03:08 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.571: INFO: At 2019-11-13 21:03:10 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.571: INFO: At 2019-11-13 21:03:13 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.571: INFO: At 2019-11-13 21:03:14 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.571: INFO: At 2019-11-13 21:03:24 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.571: INFO: At 2019-11-13 21:03:26 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.571: INFO: At 2019-11-13 21:03:26 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.571: INFO: At 2019-11-13 21:03:29 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.571: INFO: At 2019-11-13 21:03:32 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.571: INFO: At 2019-11-13 21:03:35 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.571: INFO: At 2019-11-13 21:03:37 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.571: INFO: At 2019-11-13 21:03:39 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.571: INFO: At 2019-11-13 21:03:43 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.571: INFO: At 2019-11-13 21:03:45 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.571: INFO: At 2019-11-13 21:03:47 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.571: INFO: At 2019-11-13 21:03:49 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.571: INFO: At 2019-11-13 21:03:50 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.571: INFO: At 2019-11-13 21:03:52 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.571: INFO: At 2019-11-13 21:03:54 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.571: INFO: At 2019-11-13 21:04:02 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.571: INFO: At 2019-11-13 21:04:07 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.571: INFO: At 2019-11-13 21:04:10 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.571: INFO: At 2019-11-13 21:04:11 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.571: INFO: At 2019-11-13 21:04:14 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.571: INFO: At 2019-11-13 21:04:18 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.571: INFO: At 2019-11-13 21:04:19 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.571: INFO: At 2019-11-13 21:04:19 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.571: INFO: At 2019-11-13 21:04:22 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.571: INFO: At 2019-11-13 21:04:25 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.571: INFO: At 2019-11-13 21:04:28 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 21:05:05.627: INFO: POD NODE PHASE GRACE CONDITIONS Nov 13 21:05:05.627: INFO: csi-hostpath-attacher-0 k8s-agentpool-24944505-vmss000000 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:00:06 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:01:14 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:01:14 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:00:06 +0000 UTC }] Nov 13 21:05:05.627: INFO: csi-hostpath-provisioner-0 k8s-agentpool-24944505-vmss000000 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:00:11 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:01:05 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:01:05 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:00:11 +0000 UTC }] Nov 13 21:05:05.627: INFO: csi-hostpath-resizer-0 k8s-agentpool-24944505-vmss000000 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:00:11 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:01:18 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:01:18 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:00:11 +0000 UTC }] Nov 13 21:05:05.628: INFO: csi-hostpathplugin-0 k8s-agentpool-24944505-vmss000000 Pending [] Nov 13 21:05:05.628: INFO: csi-snapshotter-0 k8s-agentpool-24944505-vmss000000 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:00:14 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:01:16 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:01:16 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:00:14 +0000 UTC }] Nov 13 21:05:05.628: INFO: Nov 13 21:05:05.746: INFO: Logging node info for node k8s-agentpool-24944505-vmss000000 Nov 13 21:05:05.802: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool-24944505-vmss000000 /api/v1/nodes/k8s-agentpool-24944505-vmss000000 5bc7cf9e-80da-4e26-a2c2-20757e870d68 35158 0 2019-11-13 20:42:13 +0000 UTC <nil> <nil> map[agentpool:agentpool beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubelet_cleanup:true kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool-24944505-vmss000000 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: storageprofile:managed storagetier:Premium_LRS] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-1137":"k8s-agentpool-24944505-vmss000000","csi-hostpath-ephemeral-9135":"k8s-agentpool-24944505-vmss000000","csi-hostpath-provisioning-2585":"k8s-agentpool-24944505-vmss000000","csi-hostpath-provisioning-3303":"k8s-agentpool-24944505-vmss000000","csi-hostpath-provisioning-716":"k8s-agentpool-24944505-vmss000000","csi-hostpath-volume-expand-4561":"k8s-agentpool-24944505-vmss000000","csi-hostpath-volume-expand-9975":"k8s-agentpool-24944505-vmss000000","csi-mock-csi-mock-volumes-2863":"csi-mock-csi-mock-volumes-2863","csi-mock-csi-mock-volumes-5360":"csi-mock-csi-mock-volumes-5360","csi-mock-csi-mock-volumes-6481":"csi-mock-csi-mock-volumes-6481","csi-mock-csi-mock-volumes-8606":"csi-mock-csi-mock-volumes-8606"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-24944505-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16797569024 0} {<nil>} 16403876Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16011137024 0} {<nil>} 15635876Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 21:04:18 +0000 UTC,LastTransitionTime:2019-11-13 20:42:03 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 21:04:18 +0000 UTC,LastTransitionTime:2019-11-13 20:42:03 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 21:04:18 +0000 UTC,LastTransitionTime:2019-11-13 20:42:03 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 21:04:18 +0000 UTC,LastTransitionTime:2019-11-13 20:42:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.248.0.4,},NodeAddress{Type:Hostname,Address:k8s-agentpool-24944505-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:0a2e8938ebfc440180ed5c0b5a9edbe7,SystemUUID:8075A1F3-D764-7248-9570-A764E51D7FF2,BootID:4f2b3cbe-c61c-4c6c-93ab-18fca0df5bc5,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[quay.io/kubernetes_incubator/nfs-provisioner@sha256:df762117e3c891f2d2ddff46ecb0776ba1f9f3c44cfd7739b0683bcd7a7954a8 quay.io/kubernetes_incubator/nfs-provisioner:v2.2.2],SizeBytes:391772778,},ContainerImage{Names:[gcr.io/google-samples/gb-frontend@sha256:35cb427341429fac3df10ff74600ea73e8ec0754d78f9ce89e0b4f3d70d53ba6 gcr.io/google-samples/gb-frontend:v6],SizeBytes:373099368,},ContainerImage{Names:[k8s.gcr.io/etcd@sha256:12c2c5e5731c3bcd56e6f1c05c0f9198b6f06793fa7fca2fb43aab9622dc4afa k8s.gcr.io/etcd:3.3.15],SizeBytes:246640776,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/volume/nfs@sha256:c2ad734346f608a5f7d69cfded93c4e8094069320657bd372d12ba21dea3ea71 gcr.io/kubernetes-e2e-test-images/volume/nfs:1.0],SizeBytes:225358913,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/jessie-dnsutils@sha256:ad583e33cb284f7ef046673809b146ec4053cda19b54a85d2b180a86169715eb gcr.io/kubernetes-e2e-test-images/jessie-dnsutils:1.0],SizeBytes:195659796,},ContainerImage{Names:[httpd@sha256:addd70e4ee83f3bc9a4c1c7c41e37927ba47faf639312fc936df3afad7926f5a httpd:2.4.39-alpine],SizeBytes:126894770,},ContainerImage{Names:[httpd@sha256:eb8ccf084cf3e80eece1add239effefd171eb39adbc154d33c14260d905d4060 httpd:2.4.38-alpine],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1],SizeBytes:121711221,},ContainerImage{Names:[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0],SizeBytes:71821984,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/sample-apiserver@sha256:1bafcc6fb1aa990b487850adba9cadc020e42d7905aa8a30481182a477ba24b0 gcr.io/kubernetes-e2e-test-images/sample-apiserver:1.10],SizeBytes:61365829,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727 gcr.io/kubernetes-e2e-test-images/agnhost:2.6],SizeBytes:57345321,},ContainerImage{Names:[quay.io/k8scsi/csi-provisioner@sha256:0efcb424f1dde9b9fb11a1a14f2e48ab47e1c3f08bc3a929990dcfcb1f7ab34f quay.io/k8scsi/csi-provisioner:v1.4.0-rc1],SizeBytes:54431016,},ContainerImage{Names:[quay.io/k8scsi/csi-snapshotter@sha256:e3d3e742e32d00488fdb401045b9b1d033d7ca0ab6e760f77b24750fc95e5f70 quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1],SizeBytes:51703561,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0],SizeBytes:46226754,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:0aba670b4d9d6b2e720bbf575d733156c676b693ca26501235444490300db838 quay.io/k8scsi/csi-attacher:v1.1.0],SizeBytes:42839085,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:7d46fb6eb8b890dc546029d1565d502b4a1d974d33625c6ee2bc7991b77fc1a1 quay.io/k8scsi/csi-resizer:v0.2.0],SizeBytes:42817100,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:f315c9042e56def3c05c6b04fe79ec9da6d39ddc557ca365a76cf35964ea08b6 quay.io/k8scsi/csi-resizer:v0.1.0],SizeBytes:42623056,},ContainerImage{Names:[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1],SizeBytes:42541759,},ContainerImage{Names:[gcr.io/google-containers/debian-base@sha256:6966a0aedd7592c18ff2dd803c08bd85780ee19f5e3a2e7cf908a4cd837afcde gcr.io/google-containers/debian-base:0.4.1],SizeBytes:42323657,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nonroot@sha256:d4ede5c74517090b6686219059118ed178cf4620f5db8781b32f806bb1e7395b gcr.io/kubernetes-e2e-test-images/nonroot:1.0],SizeBytes:42321438,},ContainerImage{Names:[redis@sha256:50899ea1ceed33fa03232f3ac57578a424faa1742c1ac9c7a7bdb95cdf19b858 redis:5.0.5-alpine],SizeBytes:29331594,},ContainerImage{Names:[quay.io/k8scsi/hostpathplugin@sha256:b4826e492fc1762fceaf9726f41575ca0a4567864d3d235da874818de18039de quay.io/k8scsi/hostpathplugin:v1.2.0-rc5],SizeBytes:28761497,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7],SizeBytes:22909487,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/echoserver@sha256:e9ba514b896cdf559eef8788b66c2c3ee55f3572df617647b4b0d8b6bf81cf19 gcr.io/kubernetes-e2e-test-images/echoserver:2.2],SizeBytes:21692741,},ContainerImage{Names:[quay.io/k8scsi/mock-driver@sha256:e0eed916b7d970bad2b7d9875f9ad16932f987f0f3d91ec5d86da68b0b5cc9d1 quay.io/k8scsi/mock-driver:v2.1.0],SizeBytes:16226335,},ContainerImage{Names:[nginx@sha256:485b610fefec7ff6c463ced9623314a04ed67e3945b9c08d7e53a47f6d108dc7 nginx:1.14-alpine],SizeBytes:16032814,},ContainerImage{Names:[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0],SizeBytes:15815995,},ContainerImage{Names:[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0],SizeBytes:14967303,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/dnsutils@sha256:2abeee84efb79c14d731966e034af33bf324d3b26ca28497555511ff094b3ddd gcr.io/kubernetes-e2e-test-images/dnsutils:1.1],SizeBytes:9349974,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nonewprivs@sha256:10066e9039219449fe3c81f38fe01928f87914150768ab81b62a468e51fa7411 gcr.io/kubernetes-e2e-test-images/nonewprivs:1.0],SizeBytes:6757579,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nautilus@sha256:33a732d4c42a266912a5091598a0f07653c9134db4b8d571690d8afd509e0bfc gcr.io/kubernetes-e2e-test-images/nautilus:1.0],SizeBytes:4753501,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/kitten@sha256:bcbc4875c982ab39aa7c4f6acf4a287f604e996d9f34a3fbda8c3d1a7457d1f6 gcr.io/kubernetes-e2e-test-images/kitten:1.0],SizeBytes:4747037,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/test-webserver@sha256:7f93d6e32798ff28bc6289254d0c2867fe2c849c8e46edc50f8624734309812e gcr.io/kubernetes-e2e-test-images/test-webserver:1.0],SizeBytes:4732240,},ContainerImage{Names:[alpine@sha256:8421d9a84432575381bfabd248f1eb56f3aa21d9d7cd2511583c68c9b7511d10 alpine:3.7],SizeBytes:4206494,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 gcr.io/kubernetes-e2e-test-images/mounttest:1.0],SizeBytes:1563521,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest-user@sha256:17319ca525ee003681fccf7e8c6b1b910ff4f49b653d939ac7f9b6e7c463933d gcr.io/kubernetes-e2e-test-images/mounttest-user:1.0],SizeBytes:1450451,},ContainerImage{Names:[busybox@sha256:1303dbf110c57f3edf68d9f5a16c082ec06c4cf7604831669faf2c712260b5a0 busybox:latest],SizeBytes:1219790,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea k8s.gcr.io/pause-amd64:3.1 k8s.gcr.io/pause:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 21:05:05.803: INFO: Logging kubelet events for node k8s-agentpool-24944505-vmss000000 Nov 13 21:05:05.863: INFO: Logging pods the kubelet thinks is on node k8s-agentpool-24944505-vmss000000 Nov 13 21:05:05.986: INFO: cleanup20-1c6cf069-2a2e-4f25-a0c5-67174b6e8721-ls58j started at 2019-11-13 21:02:42 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:05.986: INFO: Container cleanup20-1c6cf069-2a2e-4f25-a0c5-67174b6e8721 ready: true, restart count 0 Nov 13 21:05:05.986: INFO: downwardapi-volume-7f69ce2d-61f6-4d7c-b5d9-eb9a490fcf01 started at 2019-11-13 21:04:33 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:05.986: INFO: Container client-container ready: false, restart count 0 Nov 13 21:05:05.986: INFO: rs-8v6pt started at 2019-11-13 21:04:55 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:05.986: INFO: Container busybox ready: false, restart count 0 Nov 13 21:05:05.986: INFO: cleanup20-1c6cf069-2a2e-4f25-a0c5-67174b6e8721-6bc6c started at 2019-11-13 21:02:44 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:05.986: INFO: Container cleanup20-1c6cf069-2a2e-4f25-a0c5-67174b6e8721 ready: true, restart count 0 Nov 13 21:05:05.986: INFO: csi-hostpath-provisioner-0 started at 2019-11-13 21:00:11 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:05.986: INFO: Container csi-provisioner ready: true, restart count 0 Nov 13 21:05:05.986: INFO: csi-hostpath-attacher-0 started at 2019-11-13 20:58:45 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:05.986: INFO: Container csi-attacher ready: true, restart count 0 Nov 13 21:05:05.986: INFO: cleanup20-1c6cf069-2a2e-4f25-a0c5-67174b6e8721-tnw46 started at 2019-11-13 21:02:47 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:05.986: INFO: Container cleanup20-1c6cf069-2a2e-4f25-a0c5-67174b6e8721 ready: true, restart count 0 Nov 13 21:05:05.986: INFO: dns-test-7916b121-a8e8-4a85-8b67-0109bea48045 started at 2019-11-13 21:04:45 +0000 UTC (0+3 container statuses recorded) Nov 13 21:05:05.986: INFO: Container jessie-querier ready: false, restart count 0 Nov 13 21:05:05.986: INFO: Container querier ready: false, restart count 0 Nov 13 21:05:05.986: INFO: Container webserver ready: false, restart count 0 Nov 13 21:05:05.986: INFO: pod-subpath-test-local-preprovisionedpv-7ndz started at 2019-11-13 21:04:19 +0000 UTC (1+1 container statuses recorded) Nov 13 21:05:05.986: INFO: Init container init-volume-local-preprovisionedpv-7ndz ready: true, restart count 0 Nov 13 21:05:05.986: INFO: Container test-container-subpath-local-preprovisionedpv-7ndz ready: false, restart count 0 Nov 13 21:05:05.986: INFO: cleanup20-1c6cf069-2a2e-4f25-a0c5-67174b6e8721-d2878 started at 2019-11-13 21:02:45 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:05.986: INFO: Container cleanup20-1c6cf069-2a2e-4f25-a0c5-67174b6e8721 ready: true, restart count 0 Nov 13 21:05:05.986: INFO: pod-projected-configmaps-15a8f794-a2e5-4a03-aeda-2dd454aaedbf started at 2019-11-13 21:04:47 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:05.986: INFO: Container projected-configmap-volume-test ready: false, restart count 0 Nov 13 21:05:05.986: INFO: keyvault-flexvolume-sn47m started at 2019-11-13 20:42:59 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:05.986: INFO: Container keyvault-flexvolume ready: true, restart count 0 Nov 13 21:05:05.986: INFO: blobfuse-flexvol-installer-drhkj started at 2019-11-13 20:42:59 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:05.986: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 Nov 13 21:05:05.986: INFO: liveness-d82b4b32-ca11-41ae-aaa2-64e320c5c211 started at 2019-11-13 21:04:50 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:05.986: INFO: Container liveness ready: false, restart count 0 Nov 13 21:05:05.986: INFO: csi-hostpath-provisioner-0 started at 2019-11-13 20:58:50 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:05.986: INFO: Container csi-provisioner ready: false, restart count 0 Nov 13 21:05:05.986: INFO: server-envvars-4b710ced-7c1b-4260-835f-246ded177c37 started at 2019-11-13 21:04:40 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:05.986: INFO: Container srv ready: false, restart count 0 Nov 13 21:05:05.986: INFO: netserver-0 started at 2019-11-13 21:04:45 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:05.986: INFO: Container webserver ready: false, restart count 0 Nov 13 21:05:05.986: INFO: rs-ldwc2 started at 2019-11-13 21:04:56 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:05.986: INFO: Container busybox ready: false, restart count 0 Nov 13 21:05:05.986: INFO: kube-proxy-h9n4c started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:05.986: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 21:05:05.986: INFO: kubernetes-dashboard-65966766b9-lq4lm started at 2019-11-13 20:42:59 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:05.986: INFO: Container kubernetes-dashboard ready: true, restart count 0 Nov 13 21:05:05.986: INFO: external-provisioner-mpcxp started at 2019-11-13 21:02:59 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:05.986: INFO: Container nfs-provisioner ready: true, restart count 0 Nov 13 21:05:05.986: INFO: csi-hostpath-resizer-0 started at 2019-11-13 21:00:11 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:05.986: INFO: Container csi-resizer ready: true, restart count 0 Nov 13 21:05:05.986: INFO: cleanup20-1c6cf069-2a2e-4f25-a0c5-67174b6e8721-52x8m started at 2019-11-13 21:02:49 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:05.986: INFO: Container cleanup20-1c6cf069-2a2e-4f25-a0c5-67174b6e8721 ready: true, restart count 0 Nov 13 21:05:05.986: INFO: csi-hostpath-resizer-0 started at 2019-11-13 20:58:51 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:05.986: INFO: Container csi-resizer ready: true, restart count 0 Nov 13 21:05:05.986: INFO: cleanup20-1c6cf069-2a2e-4f25-a0c5-67174b6e8721-n9mwf started at 2019-11-13 21:02:49 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:05.986: INFO: Container cleanup20-1c6cf069-2a2e-4f25-a0c5-67174b6e8721 ready: true, restart count 0 Nov 13 21:05:05.986: INFO: csi-snapshotter-0 started at 2019-11-13 21:00:14 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:05.986: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 13 21:05:05.986: INFO: metrics-server-58ff8c5ddf-r2d7f started at 2019-11-13 20:42:59 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:05.986: INFO: Container metrics-server ready: true, restart count 0 Nov 13 21:05:05.986: INFO: cleanup20-1c6cf069-2a2e-4f25-a0c5-67174b6e8721-bbfjk started at 2019-11-13 21:02:46 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:05.986: INFO: Container cleanup20-1c6cf069-2a2e-4f25-a0c5-67174b6e8721 ready: true, restart count 0 Nov 13 21:05:05.986: INFO: hostexec-k8s-agentpool-24944505-vmss000000 started at 2019-11-13 21:03:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:05.986: INFO: Container agnhost ready: true, restart count 0 Nov 13 21:05:05.986: INFO: azure-ip-masq-agent-qksp6 started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:05.986: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 21:05:05.986: INFO: tiller-deploy-7559b6b885-qxwkm started at 2019-11-13 20:42:59 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:05.986: INFO: Container tiller ready: true, restart count 0 Nov 13 21:05:05.986: INFO: rs-mxw48 started at 2019-11-13 21:04:51 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:05.986: INFO: Container busybox ready: false, restart count 0 Nov 13 21:05:05.986: INFO: csi-hostpathplugin-0 started at 2019-11-13 21:04:28 +0000 UTC (0+0 container statuses recorded) Nov 13 21:05:05.986: INFO: csi-hostpathplugin-0 started at 2019-11-13 21:02:01 +0000 UTC (0+3 container statuses recorded) Nov 13 21:05:05.986: INFO: Container hostpath ready: true, restart count 0 Nov 13 21:05:05.986: INFO: Container liveness-probe ready: true, restart count 0 Nov 13 21:05:05.986: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 13 21:05:05.986: INFO: pod-5a0a0d37-c927-421c-b07e-4942df4a10fa started at 2019-11-13 21:04:22 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:05.986: INFO: Container test-container ready: false, restart count 0 Nov 13 21:05:05.986: INFO: csi-hostpath-attacher-0 started at 2019-11-13 21:00:06 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:05.986: INFO: Container csi-attacher ready: true, restart count 0 Nov 13 21:05:05.986: INFO: csi-snapshotter-0 started at 2019-11-13 20:58:52 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:05.986: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 13 21:05:05.986: INFO: hostexec-k8s-agentpool-24944505-vmss000000 started at 2019-11-13 21:04:48 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:05.986: INFO: Container agnhost ready: false, restart count 0 Nov 13 21:05:05.986: INFO: rs-mghwh started at 2019-11-13 21:04:52 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:05.986: INFO: Container busybox ready: false, restart count 0 Nov 13 21:05:05.986: INFO: pod-subpath-test-nfs-dynamicpv-rt5d started at 2019-11-13 21:04:04 +0000 UTC (1+2 container statuses recorded) Nov 13 21:05:05.986: INFO: Init container test-init-subpath-nfs-dynamicpv-rt5d ready: true, restart count 0 Nov 13 21:05:05.986: INFO: Container test-container-subpath-nfs-dynamicpv-rt5d ready: false, restart count 0 Nov 13 21:05:05.986: INFO: Container test-container-volume-nfs-dynamicpv-rt5d ready: false, restart count 0 Nov 13 21:05:05.986: INFO: rs-6b4qm started at 2019-11-13 21:04:53 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:05.986: INFO: Container busybox ready: false, restart count 0 Nov 13 21:05:05.986: INFO: cleanup20-1c6cf069-2a2e-4f25-a0c5-67174b6e8721-dbqhg started at 2019-11-13 21:02:42 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:05.986: INFO: Container cleanup20-1c6cf069-2a2e-4f25-a0c5-67174b6e8721 ready: true, restart count 0 Nov 13 21:05:05.986: INFO: cleanup20-1c6cf069-2a2e-4f25-a0c5-67174b6e8721-llrsr started at 2019-11-13 21:02:48 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:05.986: INFO: Container cleanup20-1c6cf069-2a2e-4f25-a0c5-67174b6e8721 ready: true, restart count 0 W1113 21:05:06.043219 92638 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 21:05:07.461: INFO: Latency metrics for node k8s-agentpool-24944505-vmss000000 Nov 13 21:05:07.461: INFO: Logging node info for node k8s-agentpool-24944505-vmss000001 Nov 13 21:05:07.534: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool-24944505-vmss000001 /api/v1/nodes/k8s-agentpool-24944505-vmss000001 da675d2e-0aad-4a8f-b94d-1311fb000563 36237 0 2019-11-13 20:42:24 +0000 UTC <nil> <nil> map[agentpool:agentpool beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubelet_cleanup:true kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool-24944505-vmss000001 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: storageprofile:managed storagetier:Premium_LRS] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-351":"k8s-agentpool-24944505-vmss000001","csi-hostpath-ephemeral-5584":"k8s-agentpool-24944505-vmss000001","csi-hostpath-provisioning-1638":"k8s-agentpool-24944505-vmss000001","csi-hostpath-provisioning-225":"k8s-agentpool-24944505-vmss000001","csi-hostpath-provisioning-7099":"k8s-agentpool-24944505-vmss000001","csi-hostpath-volume-6291":"k8s-agentpool-24944505-vmss000001","csi-hostpath-volume-expand-8458":"k8s-agentpool-24944505-vmss000001","csi-mock-csi-mock-volumes-4469":"csi-mock-csi-mock-volumes-4469","csi-mock-csi-mock-volumes-4959":"csi-mock-csi-mock-volumes-4959","csi-mock-csi-mock-volumes-597":"csi-mock-csi-mock-volumes-597","csi-mock-csi-mock-volumes-8812":"csi-mock-csi-mock-volumes-8812","csi-mock-csi-mock-volumes-9088":"csi-mock-csi-mock-volumes-9088"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-24944505-vmss/virtualMachines/1,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16797569024 0} {<nil>} 16403876Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16011137024 0} {<nil>} 15635876Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 21:04:50 +0000 UTC,LastTransitionTime:2019-11-13 20:42:24 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 21:04:50 +0000 UTC,LastTransitionTime:2019-11-13 20:42:24 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 21:04:50 +0000 UTC,LastTransitionTime:2019-11-13 20:42:24 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 21:04:50 +0000 UTC,LastTransitionTime:2019-11-13 20:42:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.248.0.5,},NodeAddress{Type:Hostname,Address:k8s-agentpool-24944505-vmss000001,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:6ed41b2c4b884fccaf94748594531e06,SystemUUID:9CE5C2E3-7C34-C64A-9734-2477E19937B7,BootID:306ac3f6-a23b-423e-83c9-e3d5f3938f6c,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[quay.io/kubernetes_incubator/nfs-provisioner@sha256:df762117e3c891f2d2ddff46ecb0776ba1f9f3c44cfd7739b0683bcd7a7954a8 quay.io/kubernetes_incubator/nfs-provisioner:v2.2.2],SizeBytes:391772778,},ContainerImage{Names:[gluster/glusterdynamic-provisioner@sha256:90067cb05a7d217651e84576935934fd9dcff8e6bcdcbaa416bbf36fcd09dbd1 gluster/glusterdynamic-provisioner:v1.0],SizeBytes:373281573,},ContainerImage{Names:[gcr.io/google-samples/gb-frontend@sha256:35cb427341429fac3df10ff74600ea73e8ec0754d78f9ce89e0b4f3d70d53ba6 gcr.io/google-samples/gb-frontend:v6],SizeBytes:373099368,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/volume/nfs@sha256:c2ad734346f608a5f7d69cfded93c4e8094069320657bd372d12ba21dea3ea71 gcr.io/kubernetes-e2e-test-images/volume/nfs:1.0],SizeBytes:225358913,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/jessie-dnsutils@sha256:ad583e33cb284f7ef046673809b146ec4053cda19b54a85d2b180a86169715eb gcr.io/kubernetes-e2e-test-images/jessie-dnsutils:1.0],SizeBytes:195659796,},ContainerImage{Names:[httpd@sha256:addd70e4ee83f3bc9a4c1c7c41e37927ba47faf639312fc936df3afad7926f5a httpd:2.4.39-alpine],SizeBytes:126894770,},ContainerImage{Names:[httpd@sha256:eb8ccf084cf3e80eece1add239effefd171eb39adbc154d33c14260d905d4060 httpd:2.4.38-alpine],SizeBytes:123781643,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727 gcr.io/kubernetes-e2e-test-images/agnhost:2.6],SizeBytes:57345321,},ContainerImage{Names:[quay.io/k8scsi/csi-provisioner@sha256:0efcb424f1dde9b9fb11a1a14f2e48ab47e1c3f08bc3a929990dcfcb1f7ab34f quay.io/k8scsi/csi-provisioner:v1.4.0-rc1],SizeBytes:54431016,},ContainerImage{Names:[quay.io/k8scsi/csi-snapshotter@sha256:e3d3e742e32d00488fdb401045b9b1d033d7ca0ab6e760f77b24750fc95e5f70 quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1],SizeBytes:51703561,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0],SizeBytes:46226754,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:0aba670b4d9d6b2e720bbf575d733156c676b693ca26501235444490300db838 quay.io/k8scsi/csi-attacher:v1.1.0],SizeBytes:42839085,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:7d46fb6eb8b890dc546029d1565d502b4a1d974d33625c6ee2bc7991b77fc1a1 quay.io/k8scsi/csi-resizer:v0.2.0],SizeBytes:42817100,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:f315c9042e56def3c05c6b04fe79ec9da6d39ddc557ca365a76cf35964ea08b6 quay.io/k8scsi/csi-resizer:v0.1.0],SizeBytes:42623056,},ContainerImage{Names:[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0],SizeBytes:42488424,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nonroot@sha256:d4ede5c74517090b6686219059118ed178cf4620f5db8781b32f806bb1e7395b gcr.io/kubernetes-e2e-test-images/nonroot:1.0],SizeBytes:42321438,},ContainerImage{Names:[redis@sha256:50899ea1ceed33fa03232f3ac57578a424faa1742c1ac9c7a7bdb95cdf19b858 redis:5.0.5-alpine],SizeBytes:29331594,},ContainerImage{Names:[quay.io/k8scsi/hostpathplugin@sha256:b4826e492fc1762fceaf9726f41575ca0a4567864d3d235da874818de18039de quay.io/k8scsi/hostpathplugin:v1.2.0-rc5],SizeBytes:28761497,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7],SizeBytes:22909487,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/echoserver@sha256:e9ba514b896cdf559eef8788b66c2c3ee55f3572df617647b4b0d8b6bf81cf19 gcr.io/kubernetes-e2e-test-images/echoserver:2.2],SizeBytes:21692741,},ContainerImage{Names:[quay.io/k8scsi/mock-driver@sha256:e0eed916b7d970bad2b7d9875f9ad16932f987f0f3d91ec5d86da68b0b5cc9d1 quay.io/k8scsi/mock-driver:v2.1.0],SizeBytes:16226335,},ContainerImage{Names:[nginx@sha256:485b610fefec7ff6c463ced9623314a04ed67e3945b9c08d7e53a47f6d108dc7 nginx:1.14-alpine],SizeBytes:16032814,},ContainerImage{Names:[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0],SizeBytes:15815995,},ContainerImage{Names:[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0],SizeBytes:14967303,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/dnsutils@sha256:2abeee84efb79c14d731966e034af33bf324d3b26ca28497555511ff094b3ddd gcr.io/kubernetes-e2e-test-images/dnsutils:1.1],SizeBytes:9349974,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nautilus@sha256:33a732d4c42a266912a5091598a0f07653c9134db4b8d571690d8afd509e0bfc gcr.io/kubernetes-e2e-test-images/nautilus:1.0],SizeBytes:4753501,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/kitten@sha256:bcbc4875c982ab39aa7c4f6acf4a287f604e996d9f34a3fbda8c3d1a7457d1f6 gcr.io/kubernetes-e2e-test-images/kitten:1.0],SizeBytes:4747037,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/test-webserver@sha256:7f93d6e32798ff28bc6289254d0c2867fe2c849c8e46edc50f8624734309812e gcr.io/kubernetes-e2e-test-images/test-webserver:1.0],SizeBytes:4732240,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:4206620,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 gcr.io/kubernetes-e2e-test-images/mounttest:1.0],SizeBytes:1563521,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea k8s.gcr.io/pause-amd64:3.1 k8s.gcr.io/pause:3.1],SizeBytes:742472,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-4959^4],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-4959^4,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-4469^4,DevicePath:,},},Config:nil,},} Nov 13 21:05:07.534: INFO: Logging kubelet events for node k8s-agentpool-24944505-vmss000001 Nov 13 21:05:07.595: INFO: Logging pods the kubelet thinks is on node k8s-agentpool-24944505-vmss000001 Nov 13 21:05:07.799: INFO: rs-cnqj7 started at 2019-11-13 21:04:50 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:07.799: INFO: Container busybox ready: false, restart count 0 Nov 13 21:05:07.799: INFO: blobfuse-flexvol-installer-cpm6h started at 2019-11-13 20:42:58 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:07.799: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 Nov 13 21:05:07.799: INFO: pod-1 started at 2019-11-13 21:02:31 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:07.799: INFO: Container busybox ready: true, restart count 0 Nov 13 21:05:07.799: INFO: cleanup20-1c6cf069-2a2e-4f25-a0c5-67174b6e8721-d5w8s started at 2019-11-13 21:02:41 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:07.799: INFO: Container cleanup20-1c6cf069-2a2e-4f25-a0c5-67174b6e8721 ready: true, restart count 0 Nov 13 21:05:07.799: INFO: azure-ip-masq-agent-wwpg7 started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:07.799: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 21:05:07.799: INFO: pod-subpath-test-nfs-dynamicpv-5gc6 started at 2019-11-13 21:04:22 +0000 UTC (1+1 container statuses recorded) Nov 13 21:05:07.799: INFO: Init container init-volume-nfs-dynamicpv-5gc6 ready: false, restart count 0 Nov 13 21:05:07.799: INFO: Container test-container-subpath-nfs-dynamicpv-5gc6 ready: false, restart count 0 Nov 13 21:05:07.799: INFO: cleanup20-1c6cf069-2a2e-4f25-a0c5-67174b6e8721-rp5gd started at 2019-11-13 21:02:43 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:07.799: INFO: Container cleanup20-1c6cf069-2a2e-4f25-a0c5-67174b6e8721 ready: true, restart count 0 Nov 13 21:05:07.799: INFO: rs-sffc7 started at 2019-11-13 21:04:53 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:07.799: INFO: Container busybox ready: false, restart count 0 Nov 13 21:05:07.799: INFO: pvc-volume-tester-nlvnb started at 2019-11-13 21:05:01 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:07.799: INFO: Container volume-tester ready: false, restart count 0 Nov 13 21:05:07.799: INFO: kube-proxy-7bpf7 started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:07.800: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 21:05:07.800: INFO: pod-subpath-test-local-preprovisionedpv-c9jr started at 2019-11-13 21:02:35 +0000 UTC (2+1 container statuses recorded) Nov 13 21:05:07.800: INFO: Init container init-volume-local-preprovisionedpv-c9jr ready: true, restart count 0 Nov 13 21:05:07.800: INFO: Init container test-init-volume-local-preprovisionedpv-c9jr ready: true, restart count 0 Nov 13 21:05:07.800: INFO: Container test-container-subpath-local-preprovisionedpv-c9jr ready: false, restart count 0 Nov 13 21:05:07.800: INFO: nfs-server started at 2019-11-13 21:01:50 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:07.800: INFO: Container nfs-server ready: false, restart count 0 Nov 13 21:05:07.800: INFO: rs-n9tx2 started at 2019-11-13 21:04:54 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:07.800: INFO: Container busybox ready: false, restart count 0 Nov 13 21:05:07.800: INFO: pod-0 started at 2019-11-13 21:02:30 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:07.800: INFO: Container busybox ready: true, restart count 0 Nov 13 21:05:07.800: INFO: cleanup20-1c6cf069-2a2e-4f25-a0c5-67174b6e8721-nl4h4 started at 2019-11-13 21:02:44 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:07.800: INFO: Container cleanup20-1c6cf069-2a2e-4f25-a0c5-67174b6e8721 ready: true, restart count 0 Nov 13 21:05:07.800: INFO: cleanup20-1c6cf069-2a2e-4f25-a0c5-67174b6e8721-hdl5m started at 2019-11-13 21:02:48 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:07.800: INFO: Container cleanup20-1c6cf069-2a2e-4f25-a0c5-67174b6e8721 ready: true, restart count 0 Nov 13 21:05:07.800: INFO: cleanup20-1c6cf069-2a2e-4f25-a0c5-67174b6e8721-27cr5 started at 2019-11-13 21:02:51 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:07.800: INFO: Container cleanup20-1c6cf069-2a2e-4f25-a0c5-67174b6e8721 ready: true, restart count 0 Nov 13 21:05:07.800: INFO: csi-mockplugin-0 started at 2019-11-13 21:02:54 +0000 UTC (0+3 container statuses recorded) Nov 13 21:05:07.800: INFO: Container csi-provisioner ready: true, restart count 0 Nov 13 21:05:07.800: INFO: Container driver-registrar ready: true, restart count 0 Nov 13 21:05:07.800: INFO: Container mock ready: true, restart count 0 Nov 13 21:05:07.800: INFO: redis-master-vnd85 started at 2019-11-13 21:02:29 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:07.800: INFO: Container redis-master ready: true, restart count 0 Nov 13 21:05:07.800: INFO: csi-mockplugin-attacher-0 started at 2019-11-13 21:04:09 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:07.800: INFO: Container csi-attacher ready: true, restart count 0 Nov 13 21:05:07.800: INFO: var-expansion-b7bd3ac1-44cb-4c6c-bc92-7896bd8db321 started at 2019-11-13 21:04:21 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:07.800: INFO: Container dapi-container ready: false, restart count 0 Nov 13 21:05:07.800: INFO: cleanup20-1c6cf069-2a2e-4f25-a0c5-67174b6e8721-pj8z9 started at 2019-11-13 21:02:42 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:07.800: INFO: Container cleanup20-1c6cf069-2a2e-4f25-a0c5-67174b6e8721 ready: true, restart count 0 Nov 13 21:05:07.800: INFO: cleanup20-1c6cf069-2a2e-4f25-a0c5-67174b6e8721-764vq started at 2019-11-13 21:02:45 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:07.800: INFO: Container cleanup20-1c6cf069-2a2e-4f25-a0c5-67174b6e8721 ready: true, restart count 0 Nov 13 21:05:07.800: INFO: configmap-client started at 2019-11-13 21:03:18 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:07.800: INFO: Container configmap-client ready: true, restart count 0 Nov 13 21:05:07.800: INFO: rs-smhmv started at 2019-11-13 21:04:51 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:07.800: INFO: Container busybox ready: false, restart count 0 Nov 13 21:05:07.800: INFO: pvc-volume-tester-nbtpt started at 2019-11-13 21:04:47 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:07.800: INFO: Container volume-tester ready: false, restart count 0 Nov 13 21:05:07.800: INFO: hostexec-k8s-agentpool-24944505-vmss000001 started at 2019-11-13 21:02:23 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:07.800: INFO: Container agnhost ready: true, restart count 0 Nov 13 21:05:07.800: INFO: cleanup20-1c6cf069-2a2e-4f25-a0c5-67174b6e8721-d5zvw started at 2019-11-13 21:02:50 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:07.800: INFO: Container cleanup20-1c6cf069-2a2e-4f25-a0c5-67174b6e8721 ready: true, restart count 0 Nov 13 21:05:07.800: INFO: test-webserver-063a973c-59e0-4e62-abe0-ed65b5b8a89d started at 2019-11-13 21:02:51 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:07.800: INFO: Container test-webserver ready: true, restart count 0 Nov 13 21:05:07.800: INFO: cleanup20-1c6cf069-2a2e-4f25-a0c5-67174b6e8721-txvzm started at 2019-11-13 21:02:47 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:07.800: INFO: Container cleanup20-1c6cf069-2a2e-4f25-a0c5-67174b6e8721 ready: true, restart count 0 Nov 13 21:05:07.800: INFO: cleanup20-1c6cf069-2a2e-4f25-a0c5-67174b6e8721-t7xrs started at 2019-11-13 21:02:49 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:07.800: INFO: Container cleanup20-1c6cf069-2a2e-4f25-a0c5-67174b6e8721 ready: true, restart count 0 Nov 13 21:05:07.800: INFO: csi-mockplugin-resizer-0 started at 2019-11-13 21:02:56 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:07.800: INFO: Container csi-resizer ready: true, restart count 0 Nov 13 21:05:07.800: INFO: pod-handle-http-request started at 2019-11-13 21:04:25 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:07.800: INFO: Container pod-handle-http-request ready: false, restart count 0 Nov 13 21:05:07.800: INFO: pod-subpath-test-hostpath-hvg9 started at 2019-11-13 21:05:05 +0000 UTC (0+2 container statuses recorded) Nov 13 21:05:07.800: INFO: Container test-container-subpath-hostpath-hvg9 ready: false, restart count 0 Nov 13 21:05:07.800: INFO: Container test-container-volume-hostpath-hvg9 ready: false, restart count 0 Nov 13 21:05:07.800: INFO: pod-subpath-test-local-preprovisionedpv-rd7s started at 2019-11-13 21:04:20 +0000 UTC (2+1 container statuses recorded) Nov 13 21:05:07.800: INFO: Init container init-volume-local-preprovisionedpv-rd7s ready: true, restart count 0 Nov 13 21:05:07.800: INFO: Init container test-init-volume-local-preprovisionedpv-rd7s ready: false, restart count 0 Nov 13 21:05:07.800: INFO: Container test-container-subpath-local-preprovisionedpv-rd7s ready: false, restart count 0 Nov 13 21:05:07.800: INFO: coredns-87f5d796-jd8bc started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:07.800: INFO: Container coredns ready: true, restart count 0 Nov 13 21:05:07.800: INFO: rs-wftlk started at 2019-11-13 21:04:52 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:07.800: INFO: Container busybox ready: false, restart count 0 Nov 13 21:05:07.800: INFO: cleanup20-1c6cf069-2a2e-4f25-a0c5-67174b6e8721-xz54k started at 2019-11-13 21:02:46 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:07.800: INFO: Container cleanup20-1c6cf069-2a2e-4f25-a0c5-67174b6e8721 ready: true, restart count 0 Nov 13 21:05:07.800: INFO: hostexec-k8s-agentpool-24944505-vmss000001 started at 2019-11-13 21:01:54 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:07.800: INFO: Container agnhost ready: true, restart count 0 Nov 13 21:05:07.800: INFO: csi-mockplugin-attacher-0 started at 2019-11-13 21:02:55 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:07.800: INFO: Container csi-attacher ready: true, restart count 0 Nov 13 21:05:07.800: INFO: keyvault-flexvolume-7phpr started at 2019-11-13 20:42:58 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:07.800: INFO: Container keyvault-flexvolume ready: true, restart count 0 Nov 13 21:05:07.800: INFO: external-provisioner-gsp4x started at 2019-11-13 21:02:32 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:07.800: INFO: Container nfs-provisioner ready: true, restart count 0 Nov 13 21:05:07.800: INFO: csi-mockplugin-0 started at 2019-11-13 21:04:08 +0000 UTC (0+3 container statuses recorded) Nov 13 21:05:07.800: INFO: Container csi-provisioner ready: false, restart count 0 Nov 13 21:05:07.800: INFO: Container driver-registrar ready: false, restart count 0 Nov 13 21:05:07.800: INFO: Container mock ready: false, restart count 0 Nov 13 21:05:07.800: INFO: netserver-1 started at 2019-11-13 21:04:46 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:07.800: INFO: Container webserver ready: false, restart count 0 W1113 21:05:07.856773 92638 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 21:05:08.008: INFO: Latency metrics for node k8s-agentpool-24944505-vmss000001 Nov 13 21:05:08.008: INFO: Logging node info for node k8s-master-24944505-vmss000000 Nov 13 21:05:08.063: INFO: Node Info: &Node{ObjectMeta:{k8s-master-24944505-vmss000000 /api/v1/nodes/k8s-master-24944505-vmss000000 048fbae2-b97f-43a2-b471-398180136f12 36294 0 2019-11-13 20:42:13 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-24944505-vmss000000 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-24944505-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284887552 0} {<nil>} 7114148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6498455552 0} {<nil>} 6346148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 21:05:07 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 21:05:07 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 21:05:07 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 21:05:07 +0000 UTC,LastTransitionTime:2019-11-13 20:42:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.4,},NodeAddress{Type:Hostname,Address:k8s-master-24944505-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:7bd754c0dc8647518c3043bcae3a76cc,SystemUUID:FA5CFE99-8148-6C46-B4E6-5EA2A5DE9D2A,BootID:74f7b0a2-e85c-438e-a642-c5b32900bb28,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:3237a53b135d8692ade82eca845fd0259ecbfcbe05fad35400e7d5c2018c4abd k8sprow.azurecr.io/azure-cloud-controller-manager:1194710255657816064-e688ecb0],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 21:05:08.064: INFO: Logging kubelet events for node k8s-master-24944505-vmss000000 Nov 13 21:05:08.123: INFO: Logging pods the kubelet thinks is on node k8s-master-24944505-vmss000000 Nov 13 21:05:08.203: INFO: kube-proxy-d9jjh started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:08.203: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 21:05:08.203: INFO: cloud-controller-manager-k8s-master-24944505-vmss000000 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:08.203: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 13 21:05:08.203: INFO: kube-addon-manager-k8s-master-24944505-vmss000000 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:08.203: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 13 21:05:08.203: INFO: kube-apiserver-k8s-master-24944505-vmss000000 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:08.203: INFO: Container kube-apiserver ready: true, restart count 0 Nov 13 21:05:08.203: INFO: kube-controller-manager-k8s-master-24944505-vmss000000 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:08.203: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 13 21:05:08.203: INFO: kube-scheduler-k8s-master-24944505-vmss000000 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:08.203: INFO: Container kube-scheduler ready: true, restart count 0 Nov 13 21:05:08.203: INFO: azure-ip-masq-agent-6pnst started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:08.203: INFO: Container azure-ip-masq-agent ready: true, restart count 0 W1113 21:05:08.259855 92638 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 21:05:08.384: INFO: Latency metrics for node k8s-master-24944505-vmss000000 Nov 13 21:05:08.384: INFO: Logging node info for node k8s-master-24944505-vmss000001 Nov 13 21:05:08.441: INFO: Node Info: &Node{ObjectMeta:{k8s-master-24944505-vmss000001 /api/v1/nodes/k8s-master-24944505-vmss000001 a2619878-b119-44f9-b89f-4073bebbb56c 36258 0 2019-11-13 20:42:40 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-24944505-vmss000001 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.6.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-24944505-vmss/virtualMachines/1,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.6.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284883456 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6498451456 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:43:08 +0000 UTC,LastTransitionTime:2019-11-13 20:43:08 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 21:05:03 +0000 UTC,LastTransitionTime:2019-11-13 20:42:34 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 21:05:03 +0000 UTC,LastTransitionTime:2019-11-13 20:42:34 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 21:05:03 +0000 UTC,LastTransitionTime:2019-11-13 20:42:34 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 21:05:03 +0000 UTC,LastTransitionTime:2019-11-13 20:42:40 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.5,},NodeAddress{Type:Hostname,Address:k8s-master-24944505-vmss000001,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:c866a7c1ed64481990416e24387dd317,SystemUUID:EBE64557-2AC8-5740-9FFE-AB9F12412FBC,BootID:3dbb0e0e-0479-4821-bc3f-a8269d250e4c,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:3237a53b135d8692ade82eca845fd0259ecbfcbe05fad35400e7d5c2018c4abd k8sprow.azurecr.io/azure-cloud-controller-manager:1194710255657816064-e688ecb0],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 21:05:08.441: INFO: Logging kubelet events for node k8s-master-24944505-vmss000001 Nov 13 21:05:08.502: INFO: Logging pods the kubelet thinks is on node k8s-master-24944505-vmss000001 Nov 13 21:05:08.585: INFO: kube-addon-manager-k8s-master-24944505-vmss000001 started at 2019-11-13 20:42:28 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:08.585: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 13 21:05:08.585: INFO: kube-apiserver-k8s-master-24944505-vmss000001 started at 2019-11-13 20:42:28 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:08.585: INFO: Container kube-apiserver ready: true, restart count 0 Nov 13 21:05:08.585: INFO: kube-controller-manager-k8s-master-24944505-vmss000001 started at 2019-11-13 20:42:28 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:08.585: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 13 21:05:08.585: INFO: kube-proxy-w4bz5 started at 2019-11-13 20:42:41 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:08.585: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 21:05:08.585: INFO: azure-ip-masq-agent-bs52d started at 2019-11-13 20:42:41 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:08.585: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 21:05:08.585: INFO: kube-scheduler-k8s-master-24944505-vmss000001 started at 2019-11-13 20:42:28 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:08.585: INFO: Container kube-scheduler ready: true, restart count 0 Nov 13 21:05:08.585: INFO: cloud-controller-manager-k8s-master-24944505-vmss000001 started at 2019-11-13 20:42:28 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:08.585: INFO: Container cloud-controller-manager ready: true, restart count 0 W1113 21:05:08.641854 92638 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 21:05:08.769: INFO: Latency metrics for node k8s-master-24944505-vmss000001 Nov 13 21:05:08.769: INFO: Logging node info for node k8s-master-24944505-vmss000002 Nov 13 21:05:08.825: INFO: Node Info: &Node{ObjectMeta:{k8s-master-24944505-vmss000002 /api/v1/nodes/k8s-master-24944505-vmss000002 f3b05267-a009-4dba-9f97-4f772e48707e 36288 0 2019-11-13 20:42:13 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-24944505-vmss000002 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.4.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-24944505-vmss/virtualMachines/2,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284887552 0} {<nil>} 7114148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6498455552 0} {<nil>} 6346148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 21:05:07 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 21:05:07 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 21:05:07 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 21:05:07 +0000 UTC,LastTransitionTime:2019-11-13 20:42:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.6,},NodeAddress{Type:Hostname,Address:k8s-master-24944505-vmss000002,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:fc1777f37ced4b9c97acb9b30e276c4f,SystemUUID:E9DF19DC-D02C-714C-A7D6-0BA6CE959C55,BootID:622958a8-d45d-4edf-9266-4dbb759e2359,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:3237a53b135d8692ade82eca845fd0259ecbfcbe05fad35400e7d5c2018c4abd k8sprow.azurecr.io/azure-cloud-controller-manager:1194710255657816064-e688ecb0],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 21:05:08.825: INFO: Logging kubelet events for node k8s-master-24944505-vmss000002 Nov 13 21:05:08.883: INFO: Logging pods the kubelet thinks is on node k8s-master-24944505-vmss000002 Nov 13 21:05:08.968: INFO: azure-ip-masq-agent-6skf2 started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:08.968: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 21:05:08.968: INFO: kube-proxy-xwz78 started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:08.968: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 21:05:08.968: INFO: kube-addon-manager-k8s-master-24944505-vmss000002 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:08.968: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 13 21:05:08.968: INFO: kube-apiserver-k8s-master-24944505-vmss000002 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:08.968: INFO: Container kube-apiserver ready: true, restart count 0 Nov 13 21:05:08.968: INFO: kube-controller-manager-k8s-master-24944505-vmss000002 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:08.968: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 13 21:05:08.968: INFO: kube-scheduler-k8s-master-24944505-vmss000002 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:08.968: INFO: Container kube-scheduler ready: true, restart count 0 Nov 13 21:05:08.968: INFO: cloud-controller-manager-k8s-master-24944505-vmss000002 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:08.968: INFO: Container cloud-controller-manager ready: true, restart count 0 W1113 21:05:09.025869 92638 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 21:05:09.158: INFO: Latency metrics for node k8s-master-24944505-vmss000002 Nov 13 21:05:09.158: INFO: Logging node info for node k8s-master-24944505-vmss000003 Nov 13 21:05:09.213: INFO: Node Info: &Node{ObjectMeta:{k8s-master-24944505-vmss000003 /api/v1/nodes/k8s-master-24944505-vmss000003 888cc19e-c62d-4945-a425-97d14e38a0b8 36195 0 2019-11-13 20:42:35 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-24944505-vmss000003 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.5.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-24944505-vmss/virtualMachines/3,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.5.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284887552 0} {<nil>} 7114148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6498455552 0} {<nil>} 6346148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 21:04:59 +0000 UTC,LastTransitionTime:2019-11-13 20:42:29 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 21:04:59 +0000 UTC,LastTransitionTime:2019-11-13 20:42:29 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 21:04:59 +0000 UTC,LastTransitionTime:2019-11-13 20:42:29 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 21:04:59 +0000 UTC,LastTransitionTime:2019-11-13 20:42:35 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.7,},NodeAddress{Type:Hostname,Address:k8s-master-24944505-vmss000003,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:2da5fe5a442b4e178a2b62ba91366e4b,SystemUUID:D2D1E816-D70E-3D40-B959-B00A2B76A83A,BootID:c5929aa1-3dac-45d5-bb2c-174c73c7300e,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:3237a53b135d8692ade82eca845fd0259ecbfcbe05fad35400e7d5c2018c4abd k8sprow.azurecr.io/azure-cloud-controller-manager:1194710255657816064-e688ecb0],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 21:05:09.214: INFO: Logging kubelet events for node k8s-master-24944505-vmss000003 Nov 13 21:05:09.273: INFO: Logging pods the kubelet thinks is on node k8s-master-24944505-vmss000003 Nov 13 21:05:09.352: INFO: kube-controller-manager-k8s-master-24944505-vmss000003 started at 2019-11-13 20:42:23 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:09.352: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 13 21:05:09.352: INFO: kube-scheduler-k8s-master-24944505-vmss000003 started at 2019-11-13 20:42:23 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:09.352: INFO: Container kube-scheduler ready: true, restart count 0 Nov 13 21:05:09.352: INFO: cloud-controller-manager-k8s-master-24944505-vmss000003 started at 2019-11-13 20:42:44 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:09.352: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 13 21:05:09.352: INFO: azure-ip-masq-agent-nlqss started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:09.352: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 21:05:09.352: INFO: kube-proxy-kgh5h started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:09.352: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 21:05:09.352: INFO: kube-addon-manager-k8s-master-24944505-vmss000003 started at 2019-11-13 20:42:22 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:09.352: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 13 21:05:09.352: INFO: kube-apiserver-k8s-master-24944505-vmss000003 started at 2019-11-13 20:42:23 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:09.352: INFO: Container kube-apiserver ready: true, restart count 0 W1113 21:05:09.409662 92638 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 21:05:09.535: INFO: Latency metrics for node k8s-master-24944505-vmss000003 Nov 13 21:05:09.535: INFO: Logging node info for node k8s-master-24944505-vmss000004 Nov 13 21:05:09.595: INFO: Node Info: &Node{ObjectMeta:{k8s-master-24944505-vmss000004 /api/v1/nodes/k8s-master-24944505-vmss000004 a26d844f-0612-406b-9a65-7e2ac952a2d5 36295 0 2019-11-13 20:42:14 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-24944505-vmss000004 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-24944505-vmss/virtualMachines/4,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284887552 0} {<nil>} 7114148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6498455552 0} {<nil>} 6346148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 21:05:07 +0000 UTC,LastTransitionTime:2019-11-13 20:42:09 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 21:05:07 +0000 UTC,LastTransitionTime:2019-11-13 20:42:09 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 21:05:07 +0000 UTC,LastTransitionTime:2019-11-13 20:42:09 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 21:05:07 +0000 UTC,LastTransitionTime:2019-11-13 20:42:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.8,},NodeAddress{Type:Hostname,Address:k8s-master-24944505-vmss000004,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:2208959aca1045d9b48fce8d784c1e41,SystemUUID:E253DB3A-3A4D-5648-8169-B2E2976CFB4B,BootID:09bc6101-2da9-4c8e-8274-43db8c5fe1cf,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:3237a53b135d8692ade82eca845fd0259ecbfcbe05fad35400e7d5c2018c4abd k8sprow.azurecr.io/azure-cloud-controller-manager:1194710255657816064-e688ecb0],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 21:05:09.595: INFO: Logging kubelet events for node k8s-master-24944505-vmss000004 Nov 13 21:05:09.655: INFO: Logging pods the kubelet thinks is on node k8s-master-24944505-vmss000004 Nov 13 21:05:09.737: INFO: kube-controller-manager-k8s-master-24944505-vmss000004 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:09.737: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 13 21:05:09.737: INFO: kube-scheduler-k8s-master-24944505-vmss000004 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:09.737: INFO: Container kube-scheduler ready: true, restart count 0 Nov 13 21:05:09.737: INFO: azure-ip-masq-agent-xnw2h started at 2019-11-13 20:42:38 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:09.737: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 21:05:09.737: INFO: kube-proxy-fld6p started at 2019-11-13 20:42:38 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:09.737: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 21:05:09.737: INFO: cloud-controller-manager-k8s-master-24944505-vmss000004 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:09.737: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 13 21:05:09.737: INFO: kube-addon-manager-k8s-master-24944505-vmss000004 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:09.737: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 13 21:05:09.737: INFO: kube-apiserver-k8s-master-24944505-vmss000004 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:05:09.737: INFO: Container kube-apiserver ready: true, restart count 0 W1113 21:05:09.794728 92638 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 21:05:09.919: INFO: Latency metrics for node k8s-master-24944505-vmss000004 Nov 13 21:05:09.919: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "volume-expand-434" for this suite. Nov 13 21:06:20.150: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered Nov 13 21:06:22.043: INFO: namespace volume-expand-434 deletion completed in 1m12.066560071s
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[sig\-storage\]\sCSI\sVolumes\s\[Driver\:\scsi\-hostpath\]\s\[Testpattern\:\sDynamic\sPV\s\(default\sfs\)\]\ssubPath\sshould\sbe\sable\sto\sunmount\safter\sthe\ssubpath\sdirectory\sis\sdeleted$'
test/e2e/storage/testsuites/subpath.go:425 Nov 13 20:55:19.148: Unexpected error: <*errors.errorString | 0xc0025b0210>: { s: "PersistentVolumeClaims [csi-hostpathfkhnj] not all in phase Bound within 5m0s", } PersistentVolumeClaims [csi-hostpathfkhnj] not all in phase Bound within 5m0s occurred test/e2e/storage/testsuites/base.go:366from junit_09.xml
[BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath test/e2e/storage/testsuites/base.go:93 [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath test/e2e/framework/framework.go:151 �[1mSTEP�[0m: Creating a kubernetes client Nov 13 20:50:14.902: INFO: >>> kubeConfig: /workspace/aks880331199/kubeconfig/kubeconfig.westus2.json �[1mSTEP�[0m: Building a namespace api object, basename provisioning �[1mSTEP�[0m: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in provisioning-4791 �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace [It] should be able to unmount after the subpath directory is deleted test/e2e/storage/testsuites/subpath.go:425 �[1mSTEP�[0m: deploying csi-hostpath driver Nov 13 20:50:15.447: INFO: creating *v1.ServiceAccount: provisioning-4791/csi-attacher Nov 13 20:50:15.514: INFO: creating *v1.ClusterRole: external-attacher-runner-provisioning-4791 Nov 13 20:50:15.514: INFO: Define cluster role external-attacher-runner-provisioning-4791 Nov 13 20:50:15.594: INFO: creating *v1.ClusterRoleBinding: csi-attacher-role-provisioning-4791 Nov 13 20:50:15.650: INFO: creating *v1.Role: provisioning-4791/external-attacher-cfg-provisioning-4791 Nov 13 20:50:15.709: INFO: creating *v1.RoleBinding: provisioning-4791/csi-attacher-role-cfg Nov 13 20:50:15.773: INFO: creating *v1.ServiceAccount: provisioning-4791/csi-provisioner Nov 13 20:50:15.835: INFO: creating *v1.ClusterRole: external-provisioner-runner-provisioning-4791 Nov 13 20:50:15.835: INFO: Define cluster role external-provisioner-runner-provisioning-4791 Nov 13 20:50:15.897: INFO: creating *v1.ClusterRoleBinding: csi-provisioner-role-provisioning-4791 Nov 13 20:50:15.952: INFO: creating *v1.Role: provisioning-4791/external-provisioner-cfg-provisioning-4791 Nov 13 20:50:16.011: INFO: creating *v1.RoleBinding: provisioning-4791/csi-provisioner-role-cfg Nov 13 20:50:16.073: INFO: creating *v1.ServiceAccount: provisioning-4791/csi-snapshotter Nov 13 20:50:16.152: INFO: creating *v1.ClusterRole: external-snapshotter-runner-provisioning-4791 Nov 13 20:50:16.152: INFO: Define cluster role external-snapshotter-runner-provisioning-4791 Nov 13 20:50:16.211: INFO: creating *v1.ClusterRoleBinding: csi-snapshotter-role-provisioning-4791 Nov 13 20:50:16.269: INFO: creating *v1.Role: provisioning-4791/external-snapshotter-leaderelection-provisioning-4791 Nov 13 20:50:16.327: INFO: creating *v1.RoleBinding: provisioning-4791/external-snapshotter-leaderelection Nov 13 20:50:16.391: INFO: creating *v1.ServiceAccount: provisioning-4791/csi-resizer Nov 13 20:50:16.446: INFO: creating *v1.ClusterRole: external-resizer-runner-provisioning-4791 Nov 13 20:50:16.447: INFO: Define cluster role external-resizer-runner-provisioning-4791 Nov 13 20:50:16.502: INFO: creating *v1.ClusterRoleBinding: csi-resizer-role-provisioning-4791 Nov 13 20:50:16.562: INFO: creating *v1.Role: provisioning-4791/external-resizer-cfg-provisioning-4791 Nov 13 20:50:16.624: INFO: creating *v1.RoleBinding: provisioning-4791/csi-resizer-role-cfg Nov 13 20:50:16.681: INFO: creating *v1.Service: provisioning-4791/csi-hostpath-attacher Nov 13 20:50:16.756: INFO: creating *v1.StatefulSet: provisioning-4791/csi-hostpath-attacher Nov 13 20:50:16.835: INFO: creating *v1beta1.CSIDriver: csi-hostpath-provisioning-4791 Nov 13 20:50:16.898: INFO: creating *v1.Service: provisioning-4791/csi-hostpathplugin Nov 13 20:50:17.035: INFO: creating *v1.StatefulSet: provisioning-4791/csi-hostpathplugin Nov 13 20:50:17.106: INFO: creating *v1.Service: provisioning-4791/csi-hostpath-provisioner Nov 13 20:50:17.227: INFO: creating *v1.StatefulSet: provisioning-4791/csi-hostpath-provisioner Nov 13 20:50:17.286: INFO: creating *v1.Service: provisioning-4791/csi-hostpath-resizer Nov 13 20:50:17.361: INFO: creating *v1.StatefulSet: provisioning-4791/csi-hostpath-resizer Nov 13 20:50:17.430: INFO: creating *v1.Service: provisioning-4791/csi-snapshotter Nov 13 20:50:17.518: INFO: creating *v1.StatefulSet: provisioning-4791/csi-snapshotter Nov 13 20:50:17.626: INFO: creating *v1.ClusterRoleBinding: psp-csi-hostpath-role-provisioning-4791 Nov 13 20:50:17.681: INFO: Test running for native CSI Driver, not checking metrics Nov 13 20:50:17.681: INFO: Creating resource for dynamic PV �[1mSTEP�[0m: creating a StorageClass provisioning-4791-csi-hostpath-provisioning-4791-sc7mpvc �[1mSTEP�[0m: creating a claim Nov 13 20:50:17.736: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil Nov 13 20:50:17.816: INFO: Waiting up to 5m0s for PersistentVolumeClaims [csi-hostpathfkhnj] to have phase Bound Nov 13 20:50:17.880: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:50:19.938: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:50:21.997: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:50:24.049: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:50:26.101: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:50:28.155: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:50:30.210: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:50:32.262: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:50:34.321: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:50:36.376: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:50:43.562: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:50:45.634: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:50:47.710: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:50:49.764: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:50:51.821: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:50:53.873: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:50:55.926: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:50:57.979: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:51:00.041: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:51:02.098: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:51:04.151: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:51:06.207: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:51:08.260: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:51:10.312: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:51:12.368: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:51:14.421: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:51:16.474: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:51:18.527: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:51:20.580: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:51:22.632: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:51:24.686: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:51:26.737: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:51:28.791: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:51:30.842: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:51:32.895: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:51:34.947: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:51:37.013: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:51:39.075: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:51:41.128: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:51:43.180: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:51:45.243: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:51:47.305: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:51:49.377: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:51:51.429: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:51:53.482: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:51:55.542: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:51:57.637: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:51:59.689: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:52:01.744: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:52:03.797: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:52:05.848: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:52:07.911: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:52:09.987: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:52:12.060: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:52:14.113: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:52:16.168: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:52:18.221: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:52:20.285: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:52:22.337: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:52:24.389: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:52:26.443: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:52:28.496: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:52:30.557: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:52:32.615: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:52:34.673: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:52:36.725: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:52:38.779: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:52:40.848: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:52:42.899: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:52:44.951: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:52:47.004: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:52:49.058: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:52:51.113: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:52:53.209: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:52:55.261: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:52:57.321: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:52:59.374: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:53:01.432: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:53:03.484: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:53:05.607: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:53:07.659: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:53:09.713: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:53:11.765: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:53:13.822: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:53:15.875: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:53:17.927: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:53:19.979: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:53:22.043: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:53:24.095: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:53:26.152: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:53:28.205: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:53:30.258: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:53:32.310: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:53:34.369: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:53:36.425: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:53:38.498: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:53:40.553: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:53:42.605: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:53:44.657: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:53:46.708: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:53:48.787: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:53:50.839: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:53:52.891: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:53:54.943: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:53:56.995: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:53:59.049: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:54:01.102: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:54:03.155: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:54:05.209: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:54:07.261: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:54:09.323: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:54:11.375: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:54:13.428: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:54:15.491: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:54:17.543: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:54:19.603: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:54:21.664: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:54:23.725: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:54:25.777: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:54:27.829: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:54:29.883: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:54:31.939: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:54:33.991: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:54:36.059: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:54:38.111: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:54:40.166: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:54:42.219: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:54:44.271: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:54:46.329: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:54:48.386: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:54:50.438: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:54:52.491: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:54:54.543: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:54:56.600: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:54:58.668: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:55:00.721: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:55:02.774: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:55:04.827: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:55:06.889: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:55:08.940: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:55:10.992: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:55:13.044: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:55:15.096: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:55:17.148: INFO: PersistentVolumeClaim csi-hostpathfkhnj found but phase is Pending instead of Bound. Nov 13 20:55:19.148: FAIL: Unexpected error: <*errors.errorString | 0xc0025b0210>: { s: "PersistentVolumeClaims [csi-hostpathfkhnj] not all in phase Bound within 5m0s", } PersistentVolumeClaims [csi-hostpathfkhnj] not all in phase Bound within 5m0s occurred [AfterEach] [Testpattern: Dynamic PV (default fs)] subPath test/e2e/framework/framework.go:152 �[1mSTEP�[0m: Collecting events from namespace "provisioning-4791". �[1mSTEP�[0m: Found 55 events. Nov 13 20:55:19.267: INFO: At 2019-11-13 20:50:16 +0000 UTC - event for csi-hostpath-attacher: {statefulset-controller } SuccessfulCreate: create Pod csi-hostpath-attacher-0 in StatefulSet csi-hostpath-attacher successful Nov 13 20:55:19.267: INFO: At 2019-11-13 20:50:17 +0000 UTC - event for csi-hostpath-provisioner: {statefulset-controller } SuccessfulCreate: create Pod csi-hostpath-provisioner-0 in StatefulSet csi-hostpath-provisioner successful Nov 13 20:55:19.267: INFO: At 2019-11-13 20:50:17 +0000 UTC - event for csi-hostpath-resizer: {statefulset-controller } SuccessfulCreate: create Pod csi-hostpath-resizer-0 in StatefulSet csi-hostpath-resizer successful Nov 13 20:55:19.267: INFO: At 2019-11-13 20:50:17 +0000 UTC - event for csi-hostpathfkhnj: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "csi-hostpath-provisioning-4791" or manually created by system administrator Nov 13 20:55:19.267: INFO: At 2019-11-13 20:50:17 +0000 UTC - event for csi-hostpathplugin: {statefulset-controller } SuccessfulCreate: create Pod csi-hostpathplugin-0 in StatefulSet csi-hostpathplugin successful Nov 13 20:55:19.267: INFO: At 2019-11-13 20:50:17 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000001} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:55:19.267: INFO: At 2019-11-13 20:50:17 +0000 UTC - event for csi-snapshotter: {statefulset-controller } SuccessfulCreate: create Pod csi-snapshotter-0 in StatefulSet csi-snapshotter successful Nov 13 20:55:19.267: INFO: At 2019-11-13 20:50:20 +0000 UTC - event for csi-hostpath-attacher-0: {kubelet k8s-agentpool-24944505-vmss000001} Pulling: Pulling image "quay.io/k8scsi/csi-attacher:v1.2.0" Nov 13 20:55:19.267: INFO: At 2019-11-13 20:50:21 +0000 UTC - event for csi-hostpath-attacher-0: {kubelet k8s-agentpool-24944505-vmss000001} Pulled: Successfully pulled image "quay.io/k8scsi/csi-attacher:v1.2.0" Nov 13 20:55:19.267: INFO: At 2019-11-13 20:50:21 +0000 UTC - event for csi-hostpath-provisioner-0: {kubelet k8s-agentpool-24944505-vmss000001} Pulling: Pulling image "quay.io/k8scsi/csi-provisioner:v1.4.0-rc1" Nov 13 20:55:19.267: INFO: At 2019-11-13 20:50:21 +0000 UTC - event for csi-hostpath-resizer-0: {kubelet k8s-agentpool-24944505-vmss000001} Pulling: Pulling image "quay.io/k8scsi/csi-resizer:v0.2.0" Nov 13 20:55:19.267: INFO: At 2019-11-13 20:50:22 +0000 UTC - event for csi-hostpath-attacher-0: {kubelet k8s-agentpool-24944505-vmss000001} Created: Created container csi-attacher Nov 13 20:55:19.267: INFO: At 2019-11-13 20:50:22 +0000 UTC - event for csi-hostpathplugin: {statefulset-controller } RecreatingFailedPod: StatefulSet provisioning-4791/csi-hostpathplugin is recreating failed Pod csi-hostpathplugin-0 Nov 13 20:55:19.267: INFO: At 2019-11-13 20:50:22 +0000 UTC - event for csi-hostpathplugin: {statefulset-controller } SuccessfulDelete: delete Pod csi-hostpathplugin-0 in StatefulSet csi-hostpathplugin successful Nov 13 20:55:19.267: INFO: At 2019-11-13 20:50:22 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000001} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:55:19.267: INFO: At 2019-11-13 20:50:22 +0000 UTC - event for csi-snapshotter-0: {kubelet k8s-agentpool-24944505-vmss000001} Pulling: Pulling image "quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1" Nov 13 20:55:19.267: INFO: At 2019-11-13 20:50:23 +0000 UTC - event for csi-hostpath-attacher-0: {kubelet k8s-agentpool-24944505-vmss000001} Started: Started container csi-attacher Nov 13 20:55:19.267: INFO: At 2019-11-13 20:50:24 +0000 UTC - event for csi-hostpath-provisioner-0: {kubelet k8s-agentpool-24944505-vmss000001} Created: Created container csi-provisioner Nov 13 20:55:19.267: INFO: At 2019-11-13 20:50:24 +0000 UTC - event for csi-hostpath-provisioner-0: {kubelet k8s-agentpool-24944505-vmss000001} Pulled: Successfully pulled image "quay.io/k8scsi/csi-provisioner:v1.4.0-rc1" Nov 13 20:55:19.267: INFO: At 2019-11-13 20:50:25 +0000 UTC - event for csi-hostpath-provisioner-0: {kubelet k8s-agentpool-24944505-vmss000001} Started: Started container csi-provisioner Nov 13 20:55:19.267: INFO: At 2019-11-13 20:50:25 +0000 UTC - event for csi-hostpath-resizer-0: {kubelet k8s-agentpool-24944505-vmss000001} Started: Started container csi-resizer Nov 13 20:55:19.267: INFO: At 2019-11-13 20:50:25 +0000 UTC - event for csi-hostpath-resizer-0: {kubelet k8s-agentpool-24944505-vmss000001} Created: Created container csi-resizer Nov 13 20:55:19.267: INFO: At 2019-11-13 20:50:25 +0000 UTC - event for csi-hostpath-resizer-0: {kubelet k8s-agentpool-24944505-vmss000001} Pulled: Successfully pulled image "quay.io/k8scsi/csi-resizer:v0.2.0" Nov 13 20:55:19.267: INFO: At 2019-11-13 20:50:27 +0000 UTC - event for csi-snapshotter-0: {kubelet k8s-agentpool-24944505-vmss000001} Pulled: Successfully pulled image "quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1" Nov 13 20:55:19.267: INFO: At 2019-11-13 20:50:28 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000001} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:55:19.267: INFO: At 2019-11-13 20:50:28 +0000 UTC - event for csi-snapshotter-0: {kubelet k8s-agentpool-24944505-vmss000001} Started: Started container csi-snapshotter Nov 13 20:55:19.267: INFO: At 2019-11-13 20:50:28 +0000 UTC - event for csi-snapshotter-0: {kubelet k8s-agentpool-24944505-vmss000001} Created: Created container csi-snapshotter Nov 13 20:55:19.267: INFO: At 2019-11-13 20:50:44 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000001} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:55:19.267: INFO: At 2019-11-13 20:50:47 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000001} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:55:19.267: INFO: At 2019-11-13 20:50:58 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000001} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:55:19.267: INFO: At 2019-11-13 20:51:07 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000001} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:55:19.267: INFO: At 2019-11-13 20:51:21 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000001} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:55:19.267: INFO: At 2019-11-13 20:51:28 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000001} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:55:19.267: INFO: At 2019-11-13 20:51:40 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000001} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:55:19.267: INFO: At 2019-11-13 20:51:51 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000001} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:55:19.267: INFO: At 2019-11-13 20:51:56 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000001} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:55:19.267: INFO: At 2019-11-13 20:52:08 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000001} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:55:19.267: INFO: At 2019-11-13 20:52:18 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000001} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:55:19.267: INFO: At 2019-11-13 20:52:22 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000001} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:55:19.267: INFO: At 2019-11-13 20:52:31 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000001} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:55:19.267: INFO: At 2019-11-13 20:52:37 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000001} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:55:19.267: INFO: At 2019-11-13 20:52:48 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000001} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:55:19.267: INFO: At 2019-11-13 20:53:03 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000001} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:55:19.267: INFO: At 2019-11-13 20:53:09 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000001} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:55:19.267: INFO: At 2019-11-13 20:53:32 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000001} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:55:19.267: INFO: At 2019-11-13 20:53:36 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000001} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:55:19.267: INFO: At 2019-11-13 20:53:48 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000001} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:55:19.267: INFO: At 2019-11-13 20:53:59 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000001} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:55:19.267: INFO: At 2019-11-13 20:54:06 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000001} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:55:19.267: INFO: At 2019-11-13 20:54:28 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000001} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:55:19.267: INFO: At 2019-11-13 20:54:45 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000001} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:55:19.267: INFO: At 2019-11-13 20:54:50 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000001} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:55:19.267: INFO: At 2019-11-13 20:54:59 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000001} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:55:19.267: INFO: At 2019-11-13 20:55:05 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000001} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:55:19.267: INFO: At 2019-11-13 20:55:15 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000001} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:55:19.321: INFO: POD NODE PHASE GRACE CONDITIONS Nov 13 20:55:19.321: INFO: csi-hostpath-attacher-0 k8s-agentpool-24944505-vmss000001 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:50:16 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:50:25 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:50:25 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:50:16 +0000 UTC }] Nov 13 20:55:19.321: INFO: csi-hostpath-provisioner-0 k8s-agentpool-24944505-vmss000001 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:50:17 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:50:25 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:50:25 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:50:17 +0000 UTC }] Nov 13 20:55:19.321: INFO: csi-hostpath-resizer-0 k8s-agentpool-24944505-vmss000001 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:50:17 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:50:26 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:50:26 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:50:17 +0000 UTC }] Nov 13 20:55:19.322: INFO: csi-hostpathplugin-0 k8s-agentpool-24944505-vmss000001 Pending [] Nov 13 20:55:19.322: INFO: csi-snapshotter-0 k8s-agentpool-24944505-vmss000001 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:50:18 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:50:30 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:50:30 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:50:18 +0000 UTC }] Nov 13 20:55:19.322: INFO: Nov 13 20:55:19.428: INFO: Logging node info for node k8s-agentpool-24944505-vmss000000 Nov 13 20:55:19.484: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool-24944505-vmss000000 /api/v1/nodes/k8s-agentpool-24944505-vmss000000 5bc7cf9e-80da-4e26-a2c2-20757e870d68 23709 0 2019-11-13 20:42:13 +0000 UTC <nil> <nil> map[agentpool:agentpool beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool-24944505-vmss000000 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: storageprofile:managed storagetier:Premium_LRS] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-1137":"k8s-agentpool-24944505-vmss000000","csi-hostpath-ephemeral-9135":"k8s-agentpool-24944505-vmss000000","csi-hostpath-provisioning-716":"k8s-agentpool-24944505-vmss000000","csi-hostpath-volume-expand-4561":"k8s-agentpool-24944505-vmss000000","csi-mock-csi-mock-volumes-2863":"csi-mock-csi-mock-volumes-2863","csi-mock-csi-mock-volumes-5360":"csi-mock-csi-mock-volumes-5360","csi-mock-csi-mock-volumes-8606":"csi-mock-csi-mock-volumes-8606"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-24944505-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16797569024 0} {<nil>} 16403876Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16011137024 0} {<nil>} 15635876Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 20:55:06 +0000 UTC,LastTransitionTime:2019-11-13 20:42:03 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 20:55:06 +0000 UTC,LastTransitionTime:2019-11-13 20:42:03 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 20:55:06 +0000 UTC,LastTransitionTime:2019-11-13 20:42:03 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 20:55:06 +0000 UTC,LastTransitionTime:2019-11-13 20:42:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.248.0.4,},NodeAddress{Type:Hostname,Address:k8s-agentpool-24944505-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:0a2e8938ebfc440180ed5c0b5a9edbe7,SystemUUID:8075A1F3-D764-7248-9570-A764E51D7FF2,BootID:4f2b3cbe-c61c-4c6c-93ab-18fca0df5bc5,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[quay.io/kubernetes_incubator/nfs-provisioner@sha256:df762117e3c891f2d2ddff46ecb0776ba1f9f3c44cfd7739b0683bcd7a7954a8 quay.io/kubernetes_incubator/nfs-provisioner:v2.2.2],SizeBytes:391772778,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/volume/nfs@sha256:c2ad734346f608a5f7d69cfded93c4e8094069320657bd372d12ba21dea3ea71 gcr.io/kubernetes-e2e-test-images/volume/nfs:1.0],SizeBytes:225358913,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/jessie-dnsutils@sha256:ad583e33cb284f7ef046673809b146ec4053cda19b54a85d2b180a86169715eb gcr.io/kubernetes-e2e-test-images/jessie-dnsutils:1.0],SizeBytes:195659796,},ContainerImage{Names:[httpd@sha256:addd70e4ee83f3bc9a4c1c7c41e37927ba47faf639312fc936df3afad7926f5a httpd:2.4.39-alpine],SizeBytes:126894770,},ContainerImage{Names:[httpd@sha256:eb8ccf084cf3e80eece1add239effefd171eb39adbc154d33c14260d905d4060 httpd:2.4.38-alpine],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1],SizeBytes:121711221,},ContainerImage{Names:[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0],SizeBytes:71821984,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/sample-apiserver@sha256:1bafcc6fb1aa990b487850adba9cadc020e42d7905aa8a30481182a477ba24b0 gcr.io/kubernetes-e2e-test-images/sample-apiserver:1.10],SizeBytes:61365829,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727 gcr.io/kubernetes-e2e-test-images/agnhost:2.6],SizeBytes:57345321,},ContainerImage{Names:[quay.io/k8scsi/csi-provisioner@sha256:0efcb424f1dde9b9fb11a1a14f2e48ab47e1c3f08bc3a929990dcfcb1f7ab34f quay.io/k8scsi/csi-provisioner:v1.4.0-rc1],SizeBytes:54431016,},ContainerImage{Names:[quay.io/k8scsi/csi-snapshotter@sha256:e3d3e742e32d00488fdb401045b9b1d033d7ca0ab6e760f77b24750fc95e5f70 quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1],SizeBytes:51703561,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0],SizeBytes:46226754,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:0aba670b4d9d6b2e720bbf575d733156c676b693ca26501235444490300db838 quay.io/k8scsi/csi-attacher:v1.1.0],SizeBytes:42839085,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:7d46fb6eb8b890dc546029d1565d502b4a1d974d33625c6ee2bc7991b77fc1a1 quay.io/k8scsi/csi-resizer:v0.2.0],SizeBytes:42817100,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:f315c9042e56def3c05c6b04fe79ec9da6d39ddc557ca365a76cf35964ea08b6 quay.io/k8scsi/csi-resizer:v0.1.0],SizeBytes:42623056,},ContainerImage{Names:[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1],SizeBytes:42541759,},ContainerImage{Names:[gcr.io/google-containers/debian-base@sha256:6966a0aedd7592c18ff2dd803c08bd85780ee19f5e3a2e7cf908a4cd837afcde gcr.io/google-containers/debian-base:0.4.1],SizeBytes:42323657,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nonroot@sha256:d4ede5c74517090b6686219059118ed178cf4620f5db8781b32f806bb1e7395b gcr.io/kubernetes-e2e-test-images/nonroot:1.0],SizeBytes:42321438,},ContainerImage{Names:[redis@sha256:50899ea1ceed33fa03232f3ac57578a424faa1742c1ac9c7a7bdb95cdf19b858 redis:5.0.5-alpine],SizeBytes:29331594,},ContainerImage{Names:[quay.io/k8scsi/hostpathplugin@sha256:b4826e492fc1762fceaf9726f41575ca0a4567864d3d235da874818de18039de quay.io/k8scsi/hostpathplugin:v1.2.0-rc5],SizeBytes:28761497,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7],SizeBytes:22909487,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/echoserver@sha256:e9ba514b896cdf559eef8788b66c2c3ee55f3572df617647b4b0d8b6bf81cf19 gcr.io/kubernetes-e2e-test-images/echoserver:2.2],SizeBytes:21692741,},ContainerImage{Names:[quay.io/k8scsi/mock-driver@sha256:e0eed916b7d970bad2b7d9875f9ad16932f987f0f3d91ec5d86da68b0b5cc9d1 quay.io/k8scsi/mock-driver:v2.1.0],SizeBytes:16226335,},ContainerImage{Names:[nginx@sha256:485b610fefec7ff6c463ced9623314a04ed67e3945b9c08d7e53a47f6d108dc7 nginx:1.14-alpine],SizeBytes:16032814,},ContainerImage{Names:[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0],SizeBytes:15815995,},ContainerImage{Names:[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0],SizeBytes:14967303,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/dnsutils@sha256:2abeee84efb79c14d731966e034af33bf324d3b26ca28497555511ff094b3ddd gcr.io/kubernetes-e2e-test-images/dnsutils:1.1],SizeBytes:9349974,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nonewprivs@sha256:10066e9039219449fe3c81f38fe01928f87914150768ab81b62a468e51fa7411 gcr.io/kubernetes-e2e-test-images/nonewprivs:1.0],SizeBytes:6757579,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nautilus@sha256:33a732d4c42a266912a5091598a0f07653c9134db4b8d571690d8afd509e0bfc gcr.io/kubernetes-e2e-test-images/nautilus:1.0],SizeBytes:4753501,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/kitten@sha256:bcbc4875c982ab39aa7c4f6acf4a287f604e996d9f34a3fbda8c3d1a7457d1f6 gcr.io/kubernetes-e2e-test-images/kitten:1.0],SizeBytes:4747037,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/test-webserver@sha256:7f93d6e32798ff28bc6289254d0c2867fe2c849c8e46edc50f8624734309812e gcr.io/kubernetes-e2e-test-images/test-webserver:1.0],SizeBytes:4732240,},ContainerImage{Names:[alpine@sha256:8421d9a84432575381bfabd248f1eb56f3aa21d9d7cd2511583c68c9b7511d10 alpine:3.7],SizeBytes:4206494,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 gcr.io/kubernetes-e2e-test-images/mounttest:1.0],SizeBytes:1563521,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest-user@sha256:17319ca525ee003681fccf7e8c6b1b910ff4f49b653d939ac7f9b6e7c463933d gcr.io/kubernetes-e2e-test-images/mounttest-user:1.0],SizeBytes:1450451,},ContainerImage{Names:[busybox@sha256:1303dbf110c57f3edf68d9f5a16c082ec06c4cf7604831669faf2c712260b5a0 busybox:latest],SizeBytes:1219790,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea k8s.gcr.io/pause-amd64:3.1 k8s.gcr.io/pause:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 20:55:19.484: INFO: Logging kubelet events for node k8s-agentpool-24944505-vmss000000 Nov 13 20:55:19.543: INFO: Logging pods the kubelet thinks is on node k8s-agentpool-24944505-vmss000000 Nov 13 20:55:19.661: INFO: csi-hostpath-provisioner-0 started at 2019-11-13 20:52:12 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:19.661: INFO: Container csi-provisioner ready: true, restart count 0 Nov 13 20:55:19.661: INFO: hostexec-k8s-agentpool-24944505-vmss000000 started at 2019-11-13 20:55:06 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:19.661: INFO: Container agnhost ready: true, restart count 0 Nov 13 20:55:19.661: INFO: failed-jobs-history-limit-1573678500-2vfqz started at 2019-11-13 20:55:06 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:19.661: INFO: Container c ready: false, restart count 0 Nov 13 20:55:19.661: INFO: csi-hostpath-attacher-0 started at 2019-11-13 20:52:45 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:19.661: INFO: Container csi-attacher ready: true, restart count 0 Nov 13 20:55:19.661: INFO: kube-proxy-h9n4c started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:19.661: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 20:55:19.661: INFO: kubernetes-dashboard-65966766b9-lq4lm started at 2019-11-13 20:42:59 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:19.661: INFO: Container kubernetes-dashboard ready: true, restart count 0 Nov 13 20:55:19.661: INFO: csi-hostpath-resizer-0 started at 2019-11-13 20:53:22 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:19.661: INFO: Container csi-resizer ready: false, restart count 0 Nov 13 20:55:19.661: INFO: csi-hostpath-attacher-0 started at 2019-11-13 20:52:12 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:19.661: INFO: Container csi-attacher ready: true, restart count 0 Nov 13 20:55:19.661: INFO: csi-snapshotter-0 started at 2019-11-13 20:52:46 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:19.661: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 13 20:55:19.661: INFO: csi-hostpathplugin-0 started at 2019-11-13 20:55:19 +0000 UTC (0+0 container statuses recorded) Nov 13 20:55:19.661: INFO: sample-apiserver-deployment-8447597c78-tdrxn started at 2019-11-13 20:54:41 +0000 UTC (0+2 container statuses recorded) Nov 13 20:55:19.661: INFO: Container etcd ready: true, restart count 0 Nov 13 20:55:19.661: INFO: Container sample-apiserver ready: false, restart count 0 Nov 13 20:55:19.661: INFO: hostexec-k8s-agentpool-24944505-vmss000000 started at 2019-11-13 20:55:09 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:19.661: INFO: Container agnhost ready: true, restart count 0 Nov 13 20:55:19.661: INFO: liveness-c9f46d50-17cb-4b1f-a75f-3f539c74dac7 started at 2019-11-13 20:54:45 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:19.661: INFO: Container liveness ready: true, restart count 1 Nov 13 20:55:19.661: INFO: csi-hostpathplugin-0 started at 2019-11-13 20:55:16 +0000 UTC (0+3 container statuses recorded) Nov 13 20:55:19.661: INFO: Container hostpath ready: false, restart count 0 Nov 13 20:55:19.661: INFO: Container liveness-probe ready: false, restart count 0 Nov 13 20:55:19.661: INFO: Container node-driver-registrar ready: false, restart count 0 Nov 13 20:55:19.661: INFO: csi-snapshotter-0 started at 2019-11-13 20:52:12 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:19.661: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 13 20:55:19.661: INFO: busybox-privileged-false-7bd03635-e13b-455f-b563-0bcf57d77f3b started at 2019-11-13 20:54:50 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:19.661: INFO: Container busybox-privileged-false-7bd03635-e13b-455f-b563-0bcf57d77f3b ready: false, restart count 0 Nov 13 20:55:19.661: INFO: pod-secrets-b9400895-91f4-440b-aeda-9e60ac06f45f started at 2019-11-13 20:53:32 +0000 UTC (0+3 container statuses recorded) Nov 13 20:55:19.661: INFO: Container creates-volume-test ready: true, restart count 0 Nov 13 20:55:19.661: INFO: Container dels-volume-test ready: true, restart count 0 Nov 13 20:55:19.661: INFO: Container upds-volume-test ready: true, restart count 0 Nov 13 20:55:19.661: INFO: hostexec-k8s-agentpool-24944505-vmss000000 started at 2019-11-13 20:54:49 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:19.661: INFO: Container agnhost ready: true, restart count 0 Nov 13 20:55:19.661: INFO: csi-hostpathplugin-0 started at 2019-11-13 20:54:13 +0000 UTC (0+3 container statuses recorded) Nov 13 20:55:19.661: INFO: Container hostpath ready: false, restart count 0 Nov 13 20:55:19.661: INFO: Container liveness-probe ready: false, restart count 0 Nov 13 20:55:19.661: INFO: Container node-driver-registrar ready: false, restart count 0 Nov 13 20:55:19.661: INFO: csi-hostpath-provisioner-0 started at 2019-11-13 20:53:22 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:19.661: INFO: Container csi-provisioner ready: false, restart count 0 Nov 13 20:55:19.661: INFO: metrics-server-58ff8c5ddf-r2d7f started at 2019-11-13 20:42:59 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:19.661: INFO: Container metrics-server ready: true, restart count 0 Nov 13 20:55:19.661: INFO: external-provisioner-dx4bf started at 2019-11-13 20:53:08 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:19.661: INFO: Container nfs-provisioner ready: false, restart count 0 Nov 13 20:55:19.661: INFO: pod-configmaps-625b0fc6-48d7-4f4c-bccc-7bf190cc0f61 started at 2019-11-13 20:55:06 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:19.661: INFO: Container env-test ready: false, restart count 0 Nov 13 20:55:19.661: INFO: pod-4e808a3b-d331-4def-83e0-6b65ec6fc13e started at 2019-11-13 20:55:17 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:19.661: INFO: Container test-container ready: false, restart count 0 Nov 13 20:55:19.661: INFO: csi-snapshotter-0 started at 2019-11-13 20:53:22 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:19.661: INFO: Container csi-snapshotter ready: false, restart count 0 Nov 13 20:55:19.661: INFO: csi-hostpath-resizer-0 started at 2019-11-13 20:52:46 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:19.661: INFO: Container csi-resizer ready: true, restart count 0 Nov 13 20:55:19.661: INFO: azure-ip-masq-agent-qksp6 started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:19.661: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 20:55:19.661: INFO: tiller-deploy-7559b6b885-qxwkm started at 2019-11-13 20:42:59 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:19.661: INFO: Container tiller ready: true, restart count 0 Nov 13 20:55:19.661: INFO: csi-hostpath-resizer-0 started at 2019-11-13 20:52:12 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:19.661: INFO: Container csi-resizer ready: true, restart count 0 Nov 13 20:55:19.661: INFO: execpod-noendpointsvbmwb started at 2019-11-13 20:55:06 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:19.661: INFO: Container agnhost-pause ready: false, restart count 0 Nov 13 20:55:19.661: INFO: csi-hostpath-attacher-0 started at 2019-11-13 20:53:21 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:19.661: INFO: Container csi-attacher ready: false, restart count 0 Nov 13 20:55:19.661: INFO: keyvault-flexvolume-sn47m started at 2019-11-13 20:42:59 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:19.661: INFO: Container keyvault-flexvolume ready: true, restart count 0 Nov 13 20:55:19.661: INFO: blobfuse-flexvol-installer-drhkj started at 2019-11-13 20:42:59 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:19.662: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 Nov 13 20:55:19.662: INFO: csi-hostpath-provisioner-0 started at 2019-11-13 20:52:46 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:19.662: INFO: Container csi-provisioner ready: true, restart count 0 Nov 13 20:55:19.662: INFO: netserver-1 started at 2019-11-13 20:53:28 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:19.662: INFO: Container webserver ready: true, restart count 0 Nov 13 20:55:19.662: INFO: sample-webhook-deployment-86d95b659d-tmhkr started at 2019-11-13 20:55:06 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:19.662: INFO: Container sample-webhook ready: false, restart count 0 Nov 13 20:55:19.662: INFO: pod-d0181c33-d1cc-4314-afb1-6bb26491e464 started at 2019-11-13 20:55:16 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:19.662: INFO: Container test-container ready: false, restart count 0 Nov 13 20:55:19.662: INFO: httpd started at 2019-11-13 20:55:17 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:19.662: INFO: Container httpd ready: false, restart count 0 W1113 20:55:19.714495 92647 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 20:55:20.303: INFO: Latency metrics for node k8s-agentpool-24944505-vmss000000 Nov 13 20:55:20.303: INFO: Logging node info for node k8s-agentpool-24944505-vmss000001 Nov 13 20:55:20.358: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool-24944505-vmss000001 /api/v1/nodes/k8s-agentpool-24944505-vmss000001 da675d2e-0aad-4a8f-b94d-1311fb000563 23994 0 2019-11-13 20:42:24 +0000 UTC <nil> <nil> map[agentpool:agentpool beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool-24944505-vmss000001 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: storageprofile:managed storagetier:Premium_LRS] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-5584":"k8s-agentpool-24944505-vmss000001","csi-hostpath-provisioning-1638":"k8s-agentpool-24944505-vmss000001","csi-hostpath-provisioning-225":"k8s-agentpool-24944505-vmss000001","csi-hostpath-provisioning-7099":"k8s-agentpool-24944505-vmss000001","csi-hostpath-volume-6291":"k8s-agentpool-24944505-vmss000001","csi-hostpath-volume-expand-8458":"k8s-agentpool-24944505-vmss000001","csi-mock-csi-mock-volumes-597":"csi-mock-csi-mock-volumes-597","csi-mock-csi-mock-volumes-8812":"csi-mock-csi-mock-volumes-8812","csi-mock-csi-mock-volumes-9088":"csi-mock-csi-mock-volumes-9088"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-24944505-vmss/virtualMachines/1,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16797569024 0} {<nil>} 16403876Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16011137024 0} {<nil>} 15635876Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 20:55:18 +0000 UTC,LastTransitionTime:2019-11-13 20:42:24 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 20:55:18 +0000 UTC,LastTransitionTime:2019-11-13 20:42:24 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 20:55:18 +0000 UTC,LastTransitionTime:2019-11-13 20:42:24 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 20:55:18 +0000 UTC,LastTransitionTime:2019-11-13 20:42:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.248.0.5,},NodeAddress{Type:Hostname,Address:k8s-agentpool-24944505-vmss000001,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:6ed41b2c4b884fccaf94748594531e06,SystemUUID:9CE5C2E3-7C34-C64A-9734-2477E19937B7,BootID:306ac3f6-a23b-423e-83c9-e3d5f3938f6c,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[quay.io/kubernetes_incubator/nfs-provisioner@sha256:df762117e3c891f2d2ddff46ecb0776ba1f9f3c44cfd7739b0683bcd7a7954a8 quay.io/kubernetes_incubator/nfs-provisioner:v2.2.2],SizeBytes:391772778,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/volume/nfs@sha256:c2ad734346f608a5f7d69cfded93c4e8094069320657bd372d12ba21dea3ea71 gcr.io/kubernetes-e2e-test-images/volume/nfs:1.0],SizeBytes:225358913,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/jessie-dnsutils@sha256:ad583e33cb284f7ef046673809b146ec4053cda19b54a85d2b180a86169715eb gcr.io/kubernetes-e2e-test-images/jessie-dnsutils:1.0],SizeBytes:195659796,},ContainerImage{Names:[httpd@sha256:addd70e4ee83f3bc9a4c1c7c41e37927ba47faf639312fc936df3afad7926f5a httpd:2.4.39-alpine],SizeBytes:126894770,},ContainerImage{Names:[httpd@sha256:eb8ccf084cf3e80eece1add239effefd171eb39adbc154d33c14260d905d4060 httpd:2.4.38-alpine],SizeBytes:123781643,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727 gcr.io/kubernetes-e2e-test-images/agnhost:2.6],SizeBytes:57345321,},ContainerImage{Names:[quay.io/k8scsi/csi-provisioner@sha256:0efcb424f1dde9b9fb11a1a14f2e48ab47e1c3f08bc3a929990dcfcb1f7ab34f quay.io/k8scsi/csi-provisioner:v1.4.0-rc1],SizeBytes:54431016,},ContainerImage{Names:[quay.io/k8scsi/csi-snapshotter@sha256:e3d3e742e32d00488fdb401045b9b1d033d7ca0ab6e760f77b24750fc95e5f70 quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1],SizeBytes:51703561,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0],SizeBytes:46226754,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:0aba670b4d9d6b2e720bbf575d733156c676b693ca26501235444490300db838 quay.io/k8scsi/csi-attacher:v1.1.0],SizeBytes:42839085,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:7d46fb6eb8b890dc546029d1565d502b4a1d974d33625c6ee2bc7991b77fc1a1 quay.io/k8scsi/csi-resizer:v0.2.0],SizeBytes:42817100,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:f315c9042e56def3c05c6b04fe79ec9da6d39ddc557ca365a76cf35964ea08b6 quay.io/k8scsi/csi-resizer:v0.1.0],SizeBytes:42623056,},ContainerImage{Names:[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0],SizeBytes:42488424,},ContainerImage{Names:[redis@sha256:50899ea1ceed33fa03232f3ac57578a424faa1742c1ac9c7a7bdb95cdf19b858 redis:5.0.5-alpine],SizeBytes:29331594,},ContainerImage{Names:[quay.io/k8scsi/hostpathplugin@sha256:b4826e492fc1762fceaf9726f41575ca0a4567864d3d235da874818de18039de quay.io/k8scsi/hostpathplugin:v1.2.0-rc5],SizeBytes:28761497,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7],SizeBytes:22909487,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/echoserver@sha256:e9ba514b896cdf559eef8788b66c2c3ee55f3572df617647b4b0d8b6bf81cf19 gcr.io/kubernetes-e2e-test-images/echoserver:2.2],SizeBytes:21692741,},ContainerImage{Names:[quay.io/k8scsi/mock-driver@sha256:e0eed916b7d970bad2b7d9875f9ad16932f987f0f3d91ec5d86da68b0b5cc9d1 quay.io/k8scsi/mock-driver:v2.1.0],SizeBytes:16226335,},ContainerImage{Names:[nginx@sha256:485b610fefec7ff6c463ced9623314a04ed67e3945b9c08d7e53a47f6d108dc7 nginx:1.14-alpine],SizeBytes:16032814,},ContainerImage{Names:[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0],SizeBytes:15815995,},ContainerImage{Names:[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0],SizeBytes:14967303,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/dnsutils@sha256:2abeee84efb79c14d731966e034af33bf324d3b26ca28497555511ff094b3ddd gcr.io/kubernetes-e2e-test-images/dnsutils:1.1],SizeBytes:9349974,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nautilus@sha256:33a732d4c42a266912a5091598a0f07653c9134db4b8d571690d8afd509e0bfc gcr.io/kubernetes-e2e-test-images/nautilus:1.0],SizeBytes:4753501,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/kitten@sha256:bcbc4875c982ab39aa7c4f6acf4a287f604e996d9f34a3fbda8c3d1a7457d1f6 gcr.io/kubernetes-e2e-test-images/kitten:1.0],SizeBytes:4747037,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/test-webserver@sha256:7f93d6e32798ff28bc6289254d0c2867fe2c849c8e46edc50f8624734309812e gcr.io/kubernetes-e2e-test-images/test-webserver:1.0],SizeBytes:4732240,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 gcr.io/kubernetes-e2e-test-images/mounttest:1.0],SizeBytes:1563521,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea k8s.gcr.io/pause-amd64:3.1 k8s.gcr.io/pause:3.1],SizeBytes:742472,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-8812^4],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-8812^4,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-provisioning-225^df116022-0657-11ea-8991-000d3ac2cc37,DevicePath:,},},Config:nil,},} Nov 13 20:55:20.358: INFO: Logging kubelet events for node k8s-agentpool-24944505-vmss000001 Nov 13 20:55:20.432: INFO: Logging pods the kubelet thinks is on node k8s-agentpool-24944505-vmss000001 Nov 13 20:55:20.546: INFO: downwardapi-volume-e7e1047c-2061-4d9b-9ca4-5bea33ad95de started at 2019-11-13 20:55:14 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:20.546: INFO: Container client-container ready: false, restart count 0 Nov 13 20:55:20.546: INFO: inline-volume-tester-hblrb started at 2019-11-13 20:50:19 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:20.546: INFO: Container csi-volume-tester ready: false, restart count 0 Nov 13 20:55:20.546: INFO: csi-mockplugin-0 started at 2019-11-13 20:53:56 +0000 UTC (0+3 container statuses recorded) Nov 13 20:55:20.546: INFO: Container csi-provisioner ready: true, restart count 0 Nov 13 20:55:20.546: INFO: Container driver-registrar ready: true, restart count 0 Nov 13 20:55:20.546: INFO: Container mock ready: true, restart count 0 Nov 13 20:55:20.546: INFO: csi-snapshotter-0 started at 2019-11-13 20:52:56 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:20.546: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 13 20:55:20.547: INFO: csi-hostpathplugin-0 started at 2019-11-13 20:55:15 +0000 UTC (0+0 container statuses recorded) Nov 13 20:55:20.547: INFO: csi-hostpathplugin-0 started at 2019-11-13 20:55:17 +0000 UTC (0+0 container statuses recorded) Nov 13 20:55:20.547: INFO: local-injector started at 2019-11-13 20:55:03 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:20.547: INFO: Container local-injector ready: true, restart count 0 Nov 13 20:55:20.547: INFO: pvc-volume-tester-vclb5 started at 2019-11-13 20:54:17 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:20.547: INFO: Container volume-tester ready: true, restart count 0 Nov 13 20:55:20.547: INFO: coredns-87f5d796-jd8bc started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:20.547: INFO: Container coredns ready: true, restart count 0 Nov 13 20:55:20.547: INFO: csi-hostpath-attacher-0 started at 2019-11-13 20:50:16 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:20.547: INFO: Container csi-attacher ready: true, restart count 0 Nov 13 20:55:20.547: INFO: csi-snapshotter-0 started at 2019-11-13 20:50:19 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:20.547: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 13 20:55:20.547: INFO: csi-hostpath-attacher-0 started at 2019-11-13 20:52:55 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:20.547: INFO: Container csi-attacher ready: true, restart count 0 Nov 13 20:55:20.547: INFO: csi-hostpath-provisioner-0 started at 2019-11-13 20:52:55 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:20.547: INFO: Container csi-provisioner ready: true, restart count 0 Nov 13 20:55:20.547: INFO: keyvault-flexvolume-7phpr started at 2019-11-13 20:42:58 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:20.547: INFO: Container keyvault-flexvolume ready: true, restart count 0 Nov 13 20:55:20.547: INFO: csi-hostpath-provisioner-0 started at 2019-11-13 20:50:17 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:20.547: INFO: Container csi-provisioner ready: true, restart count 0 Nov 13 20:55:20.547: INFO: csi-mockplugin-0 started at 2019-11-13 20:53:56 +0000 UTC (0+3 container statuses recorded) Nov 13 20:55:20.547: INFO: Container csi-provisioner ready: true, restart count 0 Nov 13 20:55:20.547: INFO: Container driver-registrar ready: true, restart count 0 Nov 13 20:55:20.547: INFO: Container mock ready: true, restart count 0 Nov 13 20:55:20.547: INFO: csi-hostpath-provisioner-0 started at 2019-11-13 20:50:19 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:20.547: INFO: Container csi-provisioner ready: true, restart count 0 Nov 13 20:55:20.547: INFO: csi-hostpath-resizer-0 started at 2019-11-13 20:50:19 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:20.547: INFO: Container csi-resizer ready: true, restart count 0 Nov 13 20:55:20.547: INFO: csi-mockplugin-resizer-0 started at 2019-11-13 20:53:56 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:20.547: INFO: Container csi-resizer ready: true, restart count 0 Nov 13 20:55:20.547: INFO: csi-snapshotter-0 started at 2019-11-13 20:50:18 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:20.547: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 13 20:55:20.547: INFO: blobfuse-flexvol-installer-cpm6h started at 2019-11-13 20:42:58 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:20.547: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 Nov 13 20:55:20.547: INFO: csi-mockplugin-attacher-0 started at 2019-11-13 20:53:56 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:20.547: INFO: Container csi-attacher ready: true, restart count 0 Nov 13 20:55:20.547: INFO: csi-hostpath-resizer-0 started at 2019-11-13 20:50:17 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:20.547: INFO: Container csi-resizer ready: true, restart count 0 Nov 13 20:55:20.547: INFO: csi-hostpath-attacher-0 started at 2019-11-13 20:50:18 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:20.547: INFO: Container csi-attacher ready: true, restart count 0 Nov 13 20:55:20.547: INFO: pvc-volume-tester-lwck8 started at 2019-11-13 20:54:17 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:20.547: INFO: Container volume-tester ready: true, restart count 0 Nov 13 20:55:20.547: INFO: azure-ip-masq-agent-wwpg7 started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:20.547: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 20:55:20.547: INFO: test-container-pod started at 2019-11-13 20:53:56 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:20.547: INFO: Container webserver ready: true, restart count 0 Nov 13 20:55:20.547: INFO: kube-proxy-7bpf7 started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:20.547: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 20:55:20.547: INFO: host-test-container-pod started at 2019-11-13 20:53:57 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:20.547: INFO: Container agnhost ready: true, restart count 0 Nov 13 20:55:20.547: INFO: pvc-datasource-tester-z4nbr started at 2019-11-13 20:55:06 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:20.547: INFO: Container volume-tester ready: false, restart count 0 Nov 13 20:55:20.547: INFO: csi-hostpath-resizer-0 started at 2019-11-13 20:52:56 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:20.547: INFO: Container csi-resizer ready: true, restart count 0 Nov 13 20:55:20.547: INFO: csi-hostpathplugin-0 started at 2019-11-13 20:54:39 +0000 UTC (0+3 container statuses recorded) Nov 13 20:55:20.547: INFO: Container hostpath ready: true, restart count 0 Nov 13 20:55:20.547: INFO: Container liveness-probe ready: true, restart count 0 Nov 13 20:55:20.547: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 13 20:55:20.547: INFO: hostexec-k8s-agentpool-24944505-vmss000001 started at 2019-11-13 20:54:40 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:20.547: INFO: Container agnhost ready: true, restart count 0 Nov 13 20:55:20.547: INFO: hostexec-k8s-agentpool-24944505-vmss000001 started at 2019-11-13 20:54:40 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:20.547: INFO: Container agnhost ready: true, restart count 0 W1113 20:55:20.606167 92647 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 20:55:20.784: INFO: Latency metrics for node k8s-agentpool-24944505-vmss000001 Nov 13 20:55:20.784: INFO: Logging node info for node k8s-master-24944505-vmss000000 Nov 13 20:55:20.839: INFO: Node Info: &Node{ObjectMeta:{k8s-master-24944505-vmss000000 /api/v1/nodes/k8s-master-24944505-vmss000000 048fbae2-b97f-43a2-b471-398180136f12 23642 0 2019-11-13 20:42:13 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-24944505-vmss000000 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-24944505-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284887552 0} {<nil>} 7114148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6498455552 0} {<nil>} 6346148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 20:55:06 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 20:55:06 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 20:55:06 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 20:55:06 +0000 UTC,LastTransitionTime:2019-11-13 20:42:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.4,},NodeAddress{Type:Hostname,Address:k8s-master-24944505-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:7bd754c0dc8647518c3043bcae3a76cc,SystemUUID:FA5CFE99-8148-6C46-B4E6-5EA2A5DE9D2A,BootID:74f7b0a2-e85c-438e-a642-c5b32900bb28,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:3237a53b135d8692ade82eca845fd0259ecbfcbe05fad35400e7d5c2018c4abd k8sprow.azurecr.io/azure-cloud-controller-manager:1194710255657816064-e688ecb0],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 20:55:20.840: INFO: Logging kubelet events for node k8s-master-24944505-vmss000000 Nov 13 20:55:20.896: INFO: Logging pods the kubelet thinks is on node k8s-master-24944505-vmss000000 Nov 13 20:55:20.981: INFO: kube-apiserver-k8s-master-24944505-vmss000000 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:20.982: INFO: Container kube-apiserver ready: true, restart count 0 Nov 13 20:55:20.982: INFO: kube-controller-manager-k8s-master-24944505-vmss000000 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:20.982: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 13 20:55:20.982: INFO: kube-scheduler-k8s-master-24944505-vmss000000 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:20.982: INFO: Container kube-scheduler ready: true, restart count 0 Nov 13 20:55:20.982: INFO: azure-ip-masq-agent-6pnst started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:20.982: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 20:55:20.982: INFO: kube-proxy-d9jjh started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:20.982: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 20:55:20.982: INFO: cloud-controller-manager-k8s-master-24944505-vmss000000 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:20.982: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 13 20:55:20.982: INFO: kube-addon-manager-k8s-master-24944505-vmss000000 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:20.982: INFO: Container kube-addon-manager ready: true, restart count 0 W1113 20:55:21.034927 92647 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 20:55:21.163: INFO: Latency metrics for node k8s-master-24944505-vmss000000 Nov 13 20:55:21.163: INFO: Logging node info for node k8s-master-24944505-vmss000001 Nov 13 20:55:21.215: INFO: Node Info: &Node{ObjectMeta:{k8s-master-24944505-vmss000001 /api/v1/nodes/k8s-master-24944505-vmss000001 a2619878-b119-44f9-b89f-4073bebbb56c 23467 0 2019-11-13 20:42:40 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-24944505-vmss000001 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.6.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-24944505-vmss/virtualMachines/1,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.6.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284883456 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6498451456 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:43:08 +0000 UTC,LastTransitionTime:2019-11-13 20:43:08 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 20:55:02 +0000 UTC,LastTransitionTime:2019-11-13 20:42:34 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 20:55:02 +0000 UTC,LastTransitionTime:2019-11-13 20:42:34 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 20:55:02 +0000 UTC,LastTransitionTime:2019-11-13 20:42:34 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 20:55:02 +0000 UTC,LastTransitionTime:2019-11-13 20:42:40 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.5,},NodeAddress{Type:Hostname,Address:k8s-master-24944505-vmss000001,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:c866a7c1ed64481990416e24387dd317,SystemUUID:EBE64557-2AC8-5740-9FFE-AB9F12412FBC,BootID:3dbb0e0e-0479-4821-bc3f-a8269d250e4c,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:3237a53b135d8692ade82eca845fd0259ecbfcbe05fad35400e7d5c2018c4abd k8sprow.azurecr.io/azure-cloud-controller-manager:1194710255657816064-e688ecb0],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 20:55:21.215: INFO: Logging kubelet events for node k8s-master-24944505-vmss000001 Nov 13 20:55:21.271: INFO: Logging pods the kubelet thinks is on node k8s-master-24944505-vmss000001 Nov 13 20:55:21.361: INFO: kube-scheduler-k8s-master-24944505-vmss000001 started at 2019-11-13 20:42:28 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:21.361: INFO: Container kube-scheduler ready: true, restart count 0 Nov 13 20:55:21.361: INFO: cloud-controller-manager-k8s-master-24944505-vmss000001 started at 2019-11-13 20:42:28 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:21.361: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 13 20:55:21.361: INFO: kube-addon-manager-k8s-master-24944505-vmss000001 started at 2019-11-13 20:42:28 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:21.361: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 13 20:55:21.361: INFO: kube-apiserver-k8s-master-24944505-vmss000001 started at 2019-11-13 20:42:28 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:21.361: INFO: Container kube-apiserver ready: true, restart count 0 Nov 13 20:55:21.361: INFO: kube-controller-manager-k8s-master-24944505-vmss000001 started at 2019-11-13 20:42:28 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:21.361: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 13 20:55:21.361: INFO: kube-proxy-w4bz5 started at 2019-11-13 20:42:41 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:21.361: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 20:55:21.361: INFO: azure-ip-masq-agent-bs52d started at 2019-11-13 20:42:41 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:21.361: INFO: Container azure-ip-masq-agent ready: true, restart count 0 W1113 20:55:21.414767 92647 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 20:55:21.534: INFO: Latency metrics for node k8s-master-24944505-vmss000001 Nov 13 20:55:21.534: INFO: Logging node info for node k8s-master-24944505-vmss000002 Nov 13 20:55:21.591: INFO: Node Info: &Node{ObjectMeta:{k8s-master-24944505-vmss000002 /api/v1/nodes/k8s-master-24944505-vmss000002 f3b05267-a009-4dba-9f97-4f772e48707e 23640 0 2019-11-13 20:42:13 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-24944505-vmss000002 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.4.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-24944505-vmss/virtualMachines/2,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284887552 0} {<nil>} 7114148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6498455552 0} {<nil>} 6346148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 20:55:05 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 20:55:05 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 20:55:05 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 20:55:05 +0000 UTC,LastTransitionTime:2019-11-13 20:42:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.6,},NodeAddress{Type:Hostname,Address:k8s-master-24944505-vmss000002,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:fc1777f37ced4b9c97acb9b30e276c4f,SystemUUID:E9DF19DC-D02C-714C-A7D6-0BA6CE959C55,BootID:622958a8-d45d-4edf-9266-4dbb759e2359,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:3237a53b135d8692ade82eca845fd0259ecbfcbe05fad35400e7d5c2018c4abd k8sprow.azurecr.io/azure-cloud-controller-manager:1194710255657816064-e688ecb0],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 20:55:21.591: INFO: Logging kubelet events for node k8s-master-24944505-vmss000002 Nov 13 20:55:21.706: INFO: Logging pods the kubelet thinks is on node k8s-master-24944505-vmss000002 Nov 13 20:55:21.780: INFO: kube-apiserver-k8s-master-24944505-vmss000002 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:21.780: INFO: Container kube-apiserver ready: true, restart count 0 Nov 13 20:55:21.780: INFO: kube-controller-manager-k8s-master-24944505-vmss000002 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:21.780: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 13 20:55:21.780: INFO: kube-scheduler-k8s-master-24944505-vmss000002 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:21.780: INFO: Container kube-scheduler ready: true, restart count 0 Nov 13 20:55:21.780: INFO: cloud-controller-manager-k8s-master-24944505-vmss000002 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:21.780: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 13 20:55:21.780: INFO: azure-ip-masq-agent-6skf2 started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:21.780: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 20:55:21.780: INFO: kube-proxy-xwz78 started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:21.780: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 20:55:21.780: INFO: kube-addon-manager-k8s-master-24944505-vmss000002 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:21.781: INFO: Container kube-addon-manager ready: true, restart count 0 W1113 20:55:21.846902 92647 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 20:55:21.993: INFO: Latency metrics for node k8s-master-24944505-vmss000002 Nov 13 20:55:21.993: INFO: Logging node info for node k8s-master-24944505-vmss000003 Nov 13 20:55:22.048: INFO: Node Info: &Node{ObjectMeta:{k8s-master-24944505-vmss000003 /api/v1/nodes/k8s-master-24944505-vmss000003 888cc19e-c62d-4945-a425-97d14e38a0b8 23331 0 2019-11-13 20:42:35 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-24944505-vmss000003 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.5.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-24944505-vmss/virtualMachines/3,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.5.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284887552 0} {<nil>} 7114148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6498455552 0} {<nil>} 6346148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 20:54:57 +0000 UTC,LastTransitionTime:2019-11-13 20:42:29 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 20:54:57 +0000 UTC,LastTransitionTime:2019-11-13 20:42:29 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 20:54:57 +0000 UTC,LastTransitionTime:2019-11-13 20:42:29 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 20:54:57 +0000 UTC,LastTransitionTime:2019-11-13 20:42:35 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.7,},NodeAddress{Type:Hostname,Address:k8s-master-24944505-vmss000003,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:2da5fe5a442b4e178a2b62ba91366e4b,SystemUUID:D2D1E816-D70E-3D40-B959-B00A2B76A83A,BootID:c5929aa1-3dac-45d5-bb2c-174c73c7300e,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:3237a53b135d8692ade82eca845fd0259ecbfcbe05fad35400e7d5c2018c4abd k8sprow.azurecr.io/azure-cloud-controller-manager:1194710255657816064-e688ecb0],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 20:55:22.049: INFO: Logging kubelet events for node k8s-master-24944505-vmss000003 Nov 13 20:55:22.106: INFO: Logging pods the kubelet thinks is on node k8s-master-24944505-vmss000003 Nov 13 20:55:22.181: INFO: azure-ip-masq-agent-nlqss started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:22.181: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 20:55:22.182: INFO: kube-proxy-kgh5h started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:22.182: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 20:55:22.182: INFO: kube-addon-manager-k8s-master-24944505-vmss000003 started at 2019-11-13 20:42:22 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:22.182: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 13 20:55:22.182: INFO: kube-apiserver-k8s-master-24944505-vmss000003 started at 2019-11-13 20:42:23 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:22.182: INFO: Container kube-apiserver ready: true, restart count 0 Nov 13 20:55:22.182: INFO: kube-controller-manager-k8s-master-24944505-vmss000003 started at 2019-11-13 20:42:23 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:22.182: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 13 20:55:22.182: INFO: kube-scheduler-k8s-master-24944505-vmss000003 started at 2019-11-13 20:42:23 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:22.182: INFO: Container kube-scheduler ready: true, restart count 0 Nov 13 20:55:22.182: INFO: cloud-controller-manager-k8s-master-24944505-vmss000003 started at 2019-11-13 20:42:44 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:22.182: INFO: Container cloud-controller-manager ready: true, restart count 0 W1113 20:55:22.237128 92647 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 20:55:22.365: INFO: Latency metrics for node k8s-master-24944505-vmss000003 Nov 13 20:55:22.365: INFO: Logging node info for node k8s-master-24944505-vmss000004 Nov 13 20:55:22.416: INFO: Node Info: &Node{ObjectMeta:{k8s-master-24944505-vmss000004 /api/v1/nodes/k8s-master-24944505-vmss000004 a26d844f-0612-406b-9a65-7e2ac952a2d5 23649 0 2019-11-13 20:42:14 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-24944505-vmss000004 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-24944505-vmss/virtualMachines/4,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284887552 0} {<nil>} 7114148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6498455552 0} {<nil>} 6346148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 20:55:06 +0000 UTC,LastTransitionTime:2019-11-13 20:42:09 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 20:55:06 +0000 UTC,LastTransitionTime:2019-11-13 20:42:09 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 20:55:06 +0000 UTC,LastTransitionTime:2019-11-13 20:42:09 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 20:55:06 +0000 UTC,LastTransitionTime:2019-11-13 20:42:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.8,},NodeAddress{Type:Hostname,Address:k8s-master-24944505-vmss000004,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:2208959aca1045d9b48fce8d784c1e41,SystemUUID:E253DB3A-3A4D-5648-8169-B2E2976CFB4B,BootID:09bc6101-2da9-4c8e-8274-43db8c5fe1cf,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:3237a53b135d8692ade82eca845fd0259ecbfcbe05fad35400e7d5c2018c4abd k8sprow.azurecr.io/azure-cloud-controller-manager:1194710255657816064-e688ecb0],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 20:55:22.416: INFO: Logging kubelet events for node k8s-master-24944505-vmss000004 Nov 13 20:55:22.474: INFO: Logging pods the kubelet thinks is on node k8s-master-24944505-vmss000004 Nov 13 20:55:22.550: INFO: kube-addon-manager-k8s-master-24944505-vmss000004 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:22.550: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 13 20:55:22.550: INFO: kube-apiserver-k8s-master-24944505-vmss000004 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:22.550: INFO: Container kube-apiserver ready: true, restart count 0 Nov 13 20:55:22.550: INFO: kube-controller-manager-k8s-master-24944505-vmss000004 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:22.550: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 13 20:55:22.550: INFO: kube-scheduler-k8s-master-24944505-vmss000004 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:22.550: INFO: Container kube-scheduler ready: true, restart count 0 Nov 13 20:55:22.550: INFO: azure-ip-masq-agent-xnw2h started at 2019-11-13 20:42:38 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:22.550: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 20:55:22.550: INFO: kube-proxy-fld6p started at 2019-11-13 20:42:38 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:22.550: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 20:55:22.550: INFO: cloud-controller-manager-k8s-master-24944505-vmss000004 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 20:55:22.550: INFO: Container cloud-controller-manager ready: true, restart count 0 W1113 20:55:22.603431 92647 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 20:55:22.722: INFO: Latency metrics for node k8s-master-24944505-vmss000004 Nov 13 20:55:22.722: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "provisioning-4791" for this suite. WARNING: pod log: csi-hostpath-provisioner-0/csi-provisioner: pods "csi-hostpath-provisioner-0" not found Nov 13 20:55:52.943: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered Nov 13 20:55:54.741: INFO: namespace provisioning-4791 deletion completed in 31.966177463s
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[sig\-storage\]\sCSI\sVolumes\s\[Driver\:\scsi\-hostpath\]\s\[Testpattern\:\sDynamic\sPV\s\(default\sfs\)\]\ssubPath\sshould\ssupport\sexisting\sdirectories\swhen\sreadOnly\sspecified\sin\sthe\svolumeSource$'
test/e2e/storage/testsuites/subpath.go:377 Nov 13 21:28:15.125: wait for pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99" to disappear Expected success, but got an error: <*errors.errorString | 0xc0000a10a0>: { s: "timed out waiting for the condition", } timed out waiting for the condition test/e2e/framework/pods.go:178from junit_30.xml
[BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath test/e2e/storage/testsuites/base.go:93 [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath test/e2e/framework/framework.go:151 �[1mSTEP�[0m: Creating a kubernetes client Nov 13 21:18:59.601: INFO: >>> kubeConfig: /workspace/aks880331199/kubeconfig/kubeconfig.westus2.json �[1mSTEP�[0m: Building a namespace api object, basename provisioning �[1mSTEP�[0m: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in provisioning-6430 �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace [It] should support existing directories when readOnly specified in the volumeSource test/e2e/storage/testsuites/subpath.go:377 �[1mSTEP�[0m: deploying csi-hostpath driver Nov 13 21:19:00.315: INFO: creating *v1.ServiceAccount: provisioning-6430/csi-attacher Nov 13 21:19:00.373: INFO: creating *v1.ClusterRole: external-attacher-runner-provisioning-6430 Nov 13 21:19:00.373: INFO: Define cluster role external-attacher-runner-provisioning-6430 Nov 13 21:19:00.437: INFO: creating *v1.ClusterRoleBinding: csi-attacher-role-provisioning-6430 Nov 13 21:19:00.492: INFO: creating *v1.Role: provisioning-6430/external-attacher-cfg-provisioning-6430 Nov 13 21:19:00.550: INFO: creating *v1.RoleBinding: provisioning-6430/csi-attacher-role-cfg Nov 13 21:19:00.605: INFO: creating *v1.ServiceAccount: provisioning-6430/csi-provisioner Nov 13 21:19:00.667: INFO: creating *v1.ClusterRole: external-provisioner-runner-provisioning-6430 Nov 13 21:19:00.667: INFO: Define cluster role external-provisioner-runner-provisioning-6430 Nov 13 21:19:00.723: INFO: creating *v1.ClusterRoleBinding: csi-provisioner-role-provisioning-6430 Nov 13 21:19:00.784: INFO: creating *v1.Role: provisioning-6430/external-provisioner-cfg-provisioning-6430 Nov 13 21:19:00.842: INFO: creating *v1.RoleBinding: provisioning-6430/csi-provisioner-role-cfg Nov 13 21:19:00.898: INFO: creating *v1.ServiceAccount: provisioning-6430/csi-snapshotter Nov 13 21:19:00.955: INFO: creating *v1.ClusterRole: external-snapshotter-runner-provisioning-6430 Nov 13 21:19:00.955: INFO: Define cluster role external-snapshotter-runner-provisioning-6430 Nov 13 21:19:01.019: INFO: creating *v1.ClusterRoleBinding: csi-snapshotter-role-provisioning-6430 Nov 13 21:19:01.075: INFO: creating *v1.Role: provisioning-6430/external-snapshotter-leaderelection-provisioning-6430 Nov 13 21:19:01.131: INFO: creating *v1.RoleBinding: provisioning-6430/external-snapshotter-leaderelection Nov 13 21:19:01.193: INFO: creating *v1.ServiceAccount: provisioning-6430/csi-resizer Nov 13 21:19:01.252: INFO: creating *v1.ClusterRole: external-resizer-runner-provisioning-6430 Nov 13 21:19:01.252: INFO: Define cluster role external-resizer-runner-provisioning-6430 Nov 13 21:19:01.307: INFO: creating *v1.ClusterRoleBinding: csi-resizer-role-provisioning-6430 Nov 13 21:19:01.383: INFO: creating *v1.Role: provisioning-6430/external-resizer-cfg-provisioning-6430 Nov 13 21:19:01.441: INFO: creating *v1.RoleBinding: provisioning-6430/csi-resizer-role-cfg Nov 13 21:19:01.496: INFO: creating *v1.Service: provisioning-6430/csi-hostpath-attacher Nov 13 21:19:01.583: INFO: creating *v1.StatefulSet: provisioning-6430/csi-hostpath-attacher Nov 13 21:19:01.653: INFO: creating *v1beta1.CSIDriver: csi-hostpath-provisioning-6430 Nov 13 21:19:01.725: INFO: creating *v1.Service: provisioning-6430/csi-hostpathplugin Nov 13 21:19:01.818: INFO: creating *v1.StatefulSet: provisioning-6430/csi-hostpathplugin Nov 13 21:19:01.883: INFO: creating *v1.Service: provisioning-6430/csi-hostpath-provisioner Nov 13 21:19:01.979: INFO: creating *v1.StatefulSet: provisioning-6430/csi-hostpath-provisioner Nov 13 21:19:02.039: INFO: creating *v1.Service: provisioning-6430/csi-hostpath-resizer Nov 13 21:19:02.148: INFO: creating *v1.StatefulSet: provisioning-6430/csi-hostpath-resizer Nov 13 21:19:02.210: INFO: creating *v1.Service: provisioning-6430/csi-snapshotter Nov 13 21:19:02.288: INFO: creating *v1.StatefulSet: provisioning-6430/csi-snapshotter Nov 13 21:19:02.344: INFO: creating *v1.ClusterRoleBinding: psp-csi-hostpath-role-provisioning-6430 Nov 13 21:19:02.404: INFO: Test running for native CSI Driver, not checking metrics Nov 13 21:19:02.404: INFO: Creating resource for dynamic PV �[1mSTEP�[0m: creating a StorageClass provisioning-6430-csi-hostpath-provisioning-6430-scfpgmq �[1mSTEP�[0m: creating a claim Nov 13 21:19:02.465: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil Nov 13 21:19:02.535: INFO: Waiting up to 5m0s for PersistentVolumeClaims [csi-hostpatht6bxt] to have phase Bound Nov 13 21:19:02.586: INFO: PersistentVolumeClaim csi-hostpatht6bxt found but phase is Pending instead of Bound. Nov 13 21:19:04.639: INFO: PersistentVolumeClaim csi-hostpatht6bxt found but phase is Pending instead of Bound. Nov 13 21:19:06.697: INFO: PersistentVolumeClaim csi-hostpatht6bxt found but phase is Pending instead of Bound. Nov 13 21:19:08.758: INFO: PersistentVolumeClaim csi-hostpatht6bxt found but phase is Pending instead of Bound. Nov 13 21:19:10.811: INFO: PersistentVolumeClaim csi-hostpatht6bxt found but phase is Pending instead of Bound. Nov 13 21:19:12.873: INFO: PersistentVolumeClaim csi-hostpatht6bxt found but phase is Pending instead of Bound. Nov 13 21:19:14.926: INFO: PersistentVolumeClaim csi-hostpatht6bxt found but phase is Pending instead of Bound. Nov 13 21:19:16.979: INFO: PersistentVolumeClaim csi-hostpatht6bxt found but phase is Pending instead of Bound. Nov 13 21:19:19.032: INFO: PersistentVolumeClaim csi-hostpatht6bxt found but phase is Pending instead of Bound. Nov 13 21:19:21.091: INFO: PersistentVolumeClaim csi-hostpatht6bxt found but phase is Pending instead of Bound. Nov 13 21:19:23.145: INFO: PersistentVolumeClaim csi-hostpatht6bxt found but phase is Pending instead of Bound. Nov 13 21:19:25.197: INFO: PersistentVolumeClaim csi-hostpatht6bxt found but phase is Pending instead of Bound. Nov 13 21:19:27.256: INFO: PersistentVolumeClaim csi-hostpatht6bxt found but phase is Pending instead of Bound. Nov 13 21:19:29.310: INFO: PersistentVolumeClaim csi-hostpatht6bxt found but phase is Pending instead of Bound. Nov 13 21:19:31.362: INFO: PersistentVolumeClaim csi-hostpatht6bxt found but phase is Pending instead of Bound. Nov 13 21:19:33.414: INFO: PersistentVolumeClaim csi-hostpatht6bxt found but phase is Pending instead of Bound. Nov 13 21:19:35.471: INFO: PersistentVolumeClaim csi-hostpatht6bxt found but phase is Pending instead of Bound. Nov 13 21:19:37.527: INFO: PersistentVolumeClaim csi-hostpatht6bxt found but phase is Pending instead of Bound. Nov 13 21:19:39.608: INFO: PersistentVolumeClaim csi-hostpatht6bxt found but phase is Pending instead of Bound. Nov 13 21:19:41.660: INFO: PersistentVolumeClaim csi-hostpatht6bxt found but phase is Pending instead of Bound. Nov 13 21:19:43.712: INFO: PersistentVolumeClaim csi-hostpatht6bxt found but phase is Pending instead of Bound. Nov 13 21:19:45.764: INFO: PersistentVolumeClaim csi-hostpatht6bxt found but phase is Pending instead of Bound. Nov 13 21:19:47.821: INFO: PersistentVolumeClaim csi-hostpatht6bxt found but phase is Pending instead of Bound. Nov 13 21:19:49.875: INFO: PersistentVolumeClaim csi-hostpatht6bxt found but phase is Pending instead of Bound. Nov 13 21:19:51.928: INFO: PersistentVolumeClaim csi-hostpatht6bxt found but phase is Pending instead of Bound. Nov 13 21:19:54.005: INFO: PersistentVolumeClaim csi-hostpatht6bxt found but phase is Pending instead of Bound. Nov 13 21:19:56.058: INFO: PersistentVolumeClaim csi-hostpatht6bxt found but phase is Pending instead of Bound. Nov 13 21:19:58.111: INFO: PersistentVolumeClaim csi-hostpatht6bxt found but phase is Pending instead of Bound. Nov 13 21:20:00.172: INFO: PersistentVolumeClaim csi-hostpatht6bxt found but phase is Pending instead of Bound. Nov 13 21:20:02.225: INFO: PersistentVolumeClaim csi-hostpatht6bxt found but phase is Pending instead of Bound. Nov 13 21:20:04.277: INFO: PersistentVolumeClaim csi-hostpatht6bxt found but phase is Pending instead of Bound. Nov 13 21:20:06.361: INFO: PersistentVolumeClaim csi-hostpatht6bxt found but phase is Pending instead of Bound. Nov 13 21:20:08.420: INFO: PersistentVolumeClaim csi-hostpatht6bxt found but phase is Pending instead of Bound. Nov 13 21:20:10.472: INFO: PersistentVolumeClaim csi-hostpatht6bxt found but phase is Pending instead of Bound. Nov 13 21:20:12.525: INFO: PersistentVolumeClaim csi-hostpatht6bxt found and phase=Bound (1m9.990779836s) �[1mSTEP�[0m: Creating pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 �[1mSTEP�[0m: Creating a pod to test subpath Nov 13 21:20:12.696: INFO: Waiting up to 5m0s for pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99" in namespace "provisioning-6430" to be "success or failure" Nov 13 21:20:12.749: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 52.408226ms Nov 13 21:20:14.801: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 2.104951869s Nov 13 21:20:16.855: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 4.158856252s Nov 13 21:20:18.908: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 6.211714726s Nov 13 21:20:20.961: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 8.264233186s Nov 13 21:20:23.015: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 10.318115138s Nov 13 21:20:25.067: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 12.370615446s Nov 13 21:20:27.120: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 14.423532105s Nov 13 21:20:29.172: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 16.475858954s Nov 13 21:20:31.225: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 18.528505224s Nov 13 21:20:33.278: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 20.581096333s Nov 13 21:20:35.330: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 22.633455064s Nov 13 21:20:37.386: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 24.689848273s Nov 13 21:20:39.439: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 26.742932764s Nov 13 21:20:41.492: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 28.795919143s Nov 13 21:20:43.545: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 30.848480588s Nov 13 21:20:45.597: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 32.900999313s Nov 13 21:20:47.650: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 34.953414472s Nov 13 21:20:49.703: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 37.006455066s Nov 13 21:20:51.763: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 39.066441904s Nov 13 21:20:53.815: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 41.118333786s Nov 13 21:20:55.867: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 43.170911598s Nov 13 21:20:57.919: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 45.223001862s Nov 13 21:21:00.005: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 47.308924439s Nov 13 21:21:02.060: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 49.363295053s Nov 13 21:21:04.119: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 51.422968642s Nov 13 21:21:06.172: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 53.475471632s Nov 13 21:21:08.225: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 55.528412216s Nov 13 21:21:10.277: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 57.580057655s Nov 13 21:21:12.329: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 59.632580879s Nov 13 21:21:14.389: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 1m1.692565243s Nov 13 21:21:16.441: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 1m3.744549474s Nov 13 21:21:18.494: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 1m5.797980344s Nov 13 21:21:20.548: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 1m7.851323585s Nov 13 21:21:22.600: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 1m9.903369021s Nov 13 21:21:24.652: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 1m11.955449518s Nov 13 21:21:26.706: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 1m14.009144705s Nov 13 21:21:28.758: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 1m16.061759793s Nov 13 21:21:30.812: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 1m18.115928702s Nov 13 21:21:32.864: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 1m20.167776412s Nov 13 21:21:34.918: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 1m22.221170153s Nov 13 21:21:36.970: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 1m24.273860528s Nov 13 21:21:39.026: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 1m26.329543673s Nov 13 21:21:41.082: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 1m28.385458168s Nov 13 21:21:43.135: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 1m30.438428885s Nov 13 21:21:45.189: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 1m32.492272459s Nov 13 21:21:47.241: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 1m34.544878425s Nov 13 21:21:49.294: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 1m36.597220091s Nov 13 21:21:51.347: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 1m38.65028894s Nov 13 21:21:53.399: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 1m40.702787058s Nov 13 21:21:55.452: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 1m42.755222127s Nov 13 21:21:57.526: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 1m44.829958769s Nov 13 21:21:59.606: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 1m46.909994504s Nov 13 21:22:01.658: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 1m48.962033529s Nov 13 21:22:03.711: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 1m51.014379646s Nov 13 21:22:05.764: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 1m53.067442389s Nov 13 21:22:07.817: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 1m55.120215611s Nov 13 21:22:09.869: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 1m57.172928276s Nov 13 21:22:11.954: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 1m59.257091075s Nov 13 21:22:14.006: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 2m1.309282018s Nov 13 21:22:16.058: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 2m3.361255004s Nov 13 21:22:18.110: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 2m5.4135348s Nov 13 21:22:20.164: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 2m7.467954655s Nov 13 21:22:22.218: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 2m9.521159147s Nov 13 21:22:24.272: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 2m11.575101994s Nov 13 21:22:26.324: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 2m13.627940201s Nov 13 21:22:28.377: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 2m15.680475224s Nov 13 21:22:30.430: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 2m17.733205812s Nov 13 21:22:32.483: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 2m19.786241683s Nov 13 21:22:34.536: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 2m21.839658757s Nov 13 21:22:36.589: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 2m23.892666163s Nov 13 21:22:38.642: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 2m25.945469102s Nov 13 21:22:40.694: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 2m27.997949639s Nov 13 21:22:42.747: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 2m30.050804297s Nov 13 21:22:44.800: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 2m32.103222677s Nov 13 21:22:46.853: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 2m34.156691061s Nov 13 21:22:48.906: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 2m36.209450765s Nov 13 21:22:50.959: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 2m38.262392327s Nov 13 21:22:53.011: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 2m40.314712848s Nov 13 21:22:55.063: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 2m42.366761652s Nov 13 21:22:57.116: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 2m44.419156994s Nov 13 21:22:59.171: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 2m46.475046184s Nov 13 21:23:01.224: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 2m48.527646003s Nov 13 21:23:03.276: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 2m50.579336626s Nov 13 21:23:05.329: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 2m52.632360747s Nov 13 21:23:07.381: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 2m54.684915369s Nov 13 21:23:09.435: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 2m56.738219377s Nov 13 21:23:11.488: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 2m58.791160073s Nov 13 21:23:13.540: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 3m0.843737592s Nov 13 21:23:15.598: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 3m2.901106656s Nov 13 21:23:17.650: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 3m4.953693575s Nov 13 21:23:19.703: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 3m7.006470261s Nov 13 21:23:21.756: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 3m9.059468144s Nov 13 21:23:23.809: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 3m11.112639179s Nov 13 21:23:25.862: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 3m13.165409913s Nov 13 21:23:27.914: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 3m15.217737393s Nov 13 21:23:29.967: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 3m17.270429775s Nov 13 21:23:32.019: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 3m19.322636086s Nov 13 21:23:34.072: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 3m21.375360504s Nov 13 21:23:36.125: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 3m23.428552166s Nov 13 21:23:38.185: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 3m25.488686451s Nov 13 21:23:40.238: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 3m27.541728485s Nov 13 21:23:42.291: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 3m29.594086017s Nov 13 21:23:44.343: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 3m31.64653575s Nov 13 21:23:46.395: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 3m33.698677861s Nov 13 21:23:48.448: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 3m35.751567228s Nov 13 21:23:50.501: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 3m37.804201202s Nov 13 21:23:52.553: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 3m39.85683841s Nov 13 21:23:54.607: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 3m41.910154243s Nov 13 21:23:56.659: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 3m43.962387534s Nov 13 21:23:58.712: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 3m46.015573603s Nov 13 21:24:00.764: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 3m48.067523662s Nov 13 21:24:02.817: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 3m50.120392836s Nov 13 21:24:04.870: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 3m52.173300528s Nov 13 21:24:06.922: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 3m54.225720796s Nov 13 21:24:08.975: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 3m56.278120408s Nov 13 21:24:11.027: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 3m58.330581394s Nov 13 21:24:13.080: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 4m0.383238412s Nov 13 21:24:15.133: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 4m2.436380214s Nov 13 21:24:17.187: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 4m4.490475213s Nov 13 21:24:19.240: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 4m6.54317166s Nov 13 21:24:21.294: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 4m8.597395695s Nov 13 21:24:23.346: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 4m10.649942085s Nov 13 21:24:25.399: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 4m12.702457536s Nov 13 21:24:27.453: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 4m14.756689558s Nov 13 21:24:29.509: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 4m16.812276932s Nov 13 21:24:31.608: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 4m18.911301712s Nov 13 21:24:33.660: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 4m20.96383683s Nov 13 21:24:35.713: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 4m23.016796436s Nov 13 21:24:37.765: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 4m25.068839233s Nov 13 21:24:39.818: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 4m27.121487232s Nov 13 21:24:41.870: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 4m29.173422143s Nov 13 21:24:43.922: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 4m31.225984209s Nov 13 21:24:45.975: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 4m33.278882599s Nov 13 21:24:48.034: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 4m35.337700781s Nov 13 21:24:50.088: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 4m37.391907313s Nov 13 21:24:52.141: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 4m39.444918143s Nov 13 21:24:54.195: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 4m41.498634909s Nov 13 21:24:56.248: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 4m43.551755856s Nov 13 21:24:58.301: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 4m45.604503014s Nov 13 21:25:00.354: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 4m47.657462682s Nov 13 21:25:02.407: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 4m49.710803994s Nov 13 21:25:04.459: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 4m51.762817893s Nov 13 21:25:06.512: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 4m53.815656516s Nov 13 21:25:08.564: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 4m55.867754399s Nov 13 21:25:10.617: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 4m57.920639394s Nov 13 21:25:12.669: INFO: Pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99": Phase="Pending", Reason="", readiness=false. Elapsed: 4m59.97285196s Nov 13 21:25:14.798: INFO: Failed to get logs from node "k8s-agentpool-24944505-vmss000001" pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99" container "test-container-subpath-csi-hostpath-dynamicpv-2c99": the server rejected our request for an unknown reason (get pods pod-subpath-test-csi-hostpath-dynamicpv-2c99) �[1mSTEP�[0m: delete the pod Nov 13 21:25:14.869: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:25:14.970: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:25:16.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:25:17.071: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:25:18.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:25:19.071: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:25:20.971: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:25:21.073: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:25:22.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:25:23.071: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:25:24.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:25:25.072: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:25:26.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:25:27.071: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:25:28.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:25:29.071: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:25:30.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:25:31.071: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:25:32.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:25:33.072: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:25:34.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:25:35.071: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:25:36.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:25:37.071: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:25:38.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:25:39.072: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:25:40.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:25:41.073: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:25:42.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:25:43.071: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:25:44.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:25:45.073: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:25:46.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:25:47.072: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:25:48.971: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:25:49.071: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:25:50.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:25:51.072: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:25:52.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:25:53.074: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:25:54.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:25:55.073: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:25:56.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:25:57.072: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:25:58.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:25:59.071: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:26:00.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:26:01.072: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:26:02.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:26:03.072: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:26:04.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:26:05.072: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:26:06.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:26:07.071: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:26:08.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:26:09.072: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:26:10.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:26:11.072: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:26:12.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:26:13.072: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:26:14.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:26:15.072: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:26:16.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:26:17.072: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:26:18.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:26:19.073: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:26:20.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:26:21.072: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:26:22.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:26:23.074: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:26:24.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:26:25.072: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:26:26.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:26:27.072: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:26:28.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:26:29.071: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:26:30.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:26:31.072: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:26:32.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:26:33.072: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:26:34.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:26:35.072: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:26:36.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:26:37.072: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:26:38.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:26:39.071: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:26:40.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:26:41.071: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:26:42.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:26:43.071: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:26:44.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:26:45.072: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:26:46.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:26:47.071: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:26:48.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:26:49.072: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:26:50.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:26:51.071: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:26:52.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:26:53.071: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:26:54.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:26:55.072: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:26:56.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:26:57.072: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:26:58.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:26:59.071: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:27:00.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:27:01.072: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:27:02.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:27:03.071: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:27:04.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:27:05.072: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:27:06.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:27:07.071: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:27:08.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:27:09.071: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:27:10.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:27:11.071: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:27:12.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:27:13.077: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:27:14.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:27:15.072: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:27:16.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:27:17.073: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:27:18.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:27:19.072: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:27:20.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:27:21.071: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:27:22.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:27:23.072: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:27:24.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:27:25.072: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:27:26.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:27:27.072: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:27:28.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:27:29.072: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:27:30.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:27:31.072: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:27:32.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:27:33.071: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:27:34.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:27:35.071: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:27:36.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:27:37.072: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:27:38.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:27:39.071: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:27:40.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:27:41.072: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:27:42.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:27:43.072: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:27:44.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:27:45.072: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:27:46.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:27:47.072: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:27:48.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:27:49.072: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:27:50.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:27:51.072: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:27:52.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:27:53.072: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:27:54.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:27:55.071: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:27:56.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:27:57.071: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:27:58.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:27:59.074: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:28:00.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:28:01.072: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:28:02.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:28:03.072: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:28:04.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:28:05.072: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:28:06.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:28:07.071: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:28:08.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:28:09.072: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:28:10.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:28:11.072: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:28:12.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:28:13.071: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:28:14.970: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:28:15.071: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:28:15.071: INFO: Waiting for pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 to disappear Nov 13 21:28:15.125: INFO: Pod pod-subpath-test-csi-hostpath-dynamicpv-2c99 still exists Nov 13 21:28:15.125: FAIL: wait for pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99" to disappear Expected success, but got an error: <*errors.errorString | 0xc0000a10a0>: { s: "timed out waiting for the condition", } timed out waiting for the condition �[1mSTEP�[0m: Deleting pod Nov 13 21:28:15.125: INFO: Deleting pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99" in namespace "provisioning-6430" Nov 13 21:28:15.178: INFO: Wait up to 5m0s for pod "pod-subpath-test-csi-hostpath-dynamicpv-2c99" to be fully deleted �[1mSTEP�[0m: Deleting pvc Nov 13 21:28:57.292: INFO: Deleting PersistentVolumeClaim "csi-hostpatht6bxt" Nov 13 21:28:57.360: INFO: Waiting up to 5m0s for PersistentVolume pvc-d596cfa0-6aec-4c5f-8c41-85eada654887 to get deleted Nov 13 21:28:57.414: INFO: PersistentVolume pvc-d596cfa0-6aec-4c5f-8c41-85eada654887 found and phase=Released (53.721339ms) Nov 13 21:29:02.467: INFO: PersistentVolume pvc-d596cfa0-6aec-4c5f-8c41-85eada654887 was removed �[1mSTEP�[0m: Deleting sc �[1mSTEP�[0m: uninstalling csi-hostpath driver Nov 13 21:29:02.527: INFO: deleting *v1.ServiceAccount: provisioning-6430/csi-attacher Nov 13 21:29:02.596: INFO: deleting *v1.ClusterRole: external-attacher-runner-provisioning-6430 Nov 13 21:29:02.661: INFO: deleting *v1.ClusterRoleBinding: csi-attacher-role-provisioning-6430 Nov 13 21:29:02.720: INFO: deleting *v1.Role: provisioning-6430/external-attacher-cfg-provisioning-6430 Nov 13 21:29:02.781: INFO: deleting *v1.RoleBinding: provisioning-6430/csi-attacher-role-cfg Nov 13 21:29:02.841: INFO: deleting *v1.ServiceAccount: provisioning-6430/csi-provisioner Nov 13 21:29:02.908: INFO: deleting *v1.ClusterRole: external-provisioner-runner-provisioning-6430 Nov 13 21:29:02.969: INFO: deleting *v1.ClusterRoleBinding: csi-provisioner-role-provisioning-6430 Nov 13 21:29:03.029: INFO: deleting *v1.Role: provisioning-6430/external-provisioner-cfg-provisioning-6430 Nov 13 21:29:03.096: INFO: deleting *v1.RoleBinding: provisioning-6430/csi-provisioner-role-cfg Nov 13 21:29:03.156: INFO: deleting *v1.ServiceAccount: provisioning-6430/csi-snapshotter Nov 13 21:29:03.229: INFO: deleting *v1.ClusterRole: external-snapshotter-runner-provisioning-6430 Nov 13 21:29:03.288: INFO: deleting *v1.ClusterRoleBinding: csi-snapshotter-role-provisioning-6430 Nov 13 21:29:03.348: INFO: deleting *v1.Role: provisioning-6430/external-snapshotter-leaderelection-provisioning-6430 Nov 13 21:29:03.413: INFO: deleting *v1.RoleBinding: provisioning-6430/external-snapshotter-leaderelection Nov 13 21:29:03.472: INFO: deleting *v1.ServiceAccount: provisioning-6430/csi-resizer Nov 13 21:29:03.535: INFO: deleting *v1.ClusterRole: external-resizer-runner-provisioning-6430 Nov 13 21:29:03.610: INFO: deleting *v1.ClusterRoleBinding: csi-resizer-role-provisioning-6430 Nov 13 21:29:03.673: INFO: deleting *v1.Role: provisioning-6430/external-resizer-cfg-provisioning-6430 Nov 13 21:29:03.732: INFO: deleting *v1.RoleBinding: provisioning-6430/csi-resizer-role-cfg Nov 13 21:29:03.792: INFO: deleting *v1.Service: provisioning-6430/csi-hostpath-attacher Nov 13 21:29:03.909: INFO: deleting *v1.StatefulSet: provisioning-6430/csi-hostpath-attacher Nov 13 21:29:04.008: INFO: deleting *v1beta1.CSIDriver: csi-hostpath-provisioning-6430 Nov 13 21:29:04.069: INFO: deleting *v1.Service: provisioning-6430/csi-hostpathplugin Nov 13 21:29:04.159: INFO: deleting *v1.StatefulSet: provisioning-6430/csi-hostpathplugin Nov 13 21:29:04.221: INFO: deleting *v1.Service: provisioning-6430/csi-hostpath-provisioner Nov 13 21:29:04.344: INFO: deleting *v1.StatefulSet: provisioning-6430/csi-hostpath-provisioner Nov 13 21:29:04.405: INFO: deleting *v1.Service: provisioning-6430/csi-hostpath-resizer Nov 13 21:29:04.501: INFO: deleting *v1.StatefulSet: provisioning-6430/csi-hostpath-resizer Nov 13 21:29:04.562: INFO: deleting *v1.Service: provisioning-6430/csi-snapshotter Nov 13 21:29:04.657: INFO: deleting *v1.StatefulSet: provisioning-6430/csi-snapshotter Nov 13 21:29:04.717: INFO: deleting *v1.ClusterRoleBinding: psp-csi-hostpath-role-provisioning-6430 [AfterEach] [Testpattern: Dynamic PV (default fs)] subPath test/e2e/framework/framework.go:152 �[1mSTEP�[0m: Collecting events from namespace "provisioning-6430". �[1mSTEP�[0m: Found 50 events. Nov 13 21:29:04.832: INFO: At 2019-11-13 21:19:01 +0000 UTC - event for csi-hostpath-attacher: {statefulset-controller } SuccessfulCreate: create Pod csi-hostpath-attacher-0 in StatefulSet csi-hostpath-attacher successful Nov 13 21:29:04.832: INFO: At 2019-11-13 21:19:01 +0000 UTC - event for csi-hostpathplugin: {statefulset-controller } SuccessfulCreate: create Pod csi-hostpathplugin-0 in StatefulSet csi-hostpathplugin successful Nov 13 21:29:04.832: INFO: At 2019-11-13 21:19:02 +0000 UTC - event for csi-hostpath-provisioner: {statefulset-controller } SuccessfulCreate: create Pod csi-hostpath-provisioner-0 in StatefulSet csi-hostpath-provisioner successful Nov 13 21:29:04.832: INFO: At 2019-11-13 21:19:02 +0000 UTC - event for csi-hostpath-resizer: {statefulset-controller } SuccessfulCreate: create Pod csi-hostpath-resizer-0 in StatefulSet csi-hostpath-resizer successful Nov 13 21:29:04.832: INFO: At 2019-11-13 21:19:02 +0000 UTC - event for csi-hostpatht6bxt: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "csi-hostpath-provisioning-6430" or manually created by system administrator Nov 13 21:29:04.832: INFO: At 2019-11-13 21:19:02 +0000 UTC - event for csi-snapshotter: {statefulset-controller } SuccessfulCreate: create Pod csi-snapshotter-0 in StatefulSet csi-snapshotter successful Nov 13 21:29:04.832: INFO: At 2019-11-13 21:19:34 +0000 UTC - event for csi-hostpath-provisioner-0: {kubelet k8s-agentpool-24944505-vmss000001} Pulling: Pulling image "quay.io/k8scsi/csi-provisioner:v1.4.0-rc1" Nov 13 21:29:04.832: INFO: At 2019-11-13 21:19:38 +0000 UTC - event for csi-hostpath-provisioner-0: {kubelet k8s-agentpool-24944505-vmss000001} Pulled: Successfully pulled image "quay.io/k8scsi/csi-provisioner:v1.4.0-rc1" Nov 13 21:29:04.832: INFO: At 2019-11-13 21:19:38 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000001} Pulling: Pulling image "quay.io/k8scsi/csi-node-driver-registrar:v1.1.0" Nov 13 21:29:04.832: INFO: At 2019-11-13 21:19:41 +0000 UTC - event for csi-hostpath-attacher-0: {kubelet k8s-agentpool-24944505-vmss000001} Pulling: Pulling image "quay.io/k8scsi/csi-attacher:v1.2.0" Nov 13 21:29:04.832: INFO: At 2019-11-13 21:19:41 +0000 UTC - event for csi-hostpath-provisioner-0: {kubelet k8s-agentpool-24944505-vmss000001} Created: Created container csi-provisioner Nov 13 21:29:04.832: INFO: At 2019-11-13 21:19:41 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000001} Pulled: Successfully pulled image "quay.io/k8scsi/csi-node-driver-registrar:v1.1.0" Nov 13 21:29:04.832: INFO: At 2019-11-13 21:19:43 +0000 UTC - event for csi-hostpath-attacher-0: {kubelet k8s-agentpool-24944505-vmss000001} Pulled: Successfully pulled image "quay.io/k8scsi/csi-attacher:v1.2.0" Nov 13 21:29:04.832: INFO: At 2019-11-13 21:19:45 +0000 UTC - event for csi-hostpath-resizer-0: {kubelet k8s-agentpool-24944505-vmss000001} Pulling: Pulling image "quay.io/k8scsi/csi-resizer:v0.2.0" Nov 13 21:29:04.833: INFO: At 2019-11-13 21:19:45 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000001} Created: Created container node-driver-registrar Nov 13 21:29:04.833: INFO: At 2019-11-13 21:19:48 +0000 UTC - event for csi-hostpath-attacher-0: {kubelet k8s-agentpool-24944505-vmss000001} Created: Created container csi-attacher Nov 13 21:29:04.833: INFO: At 2019-11-13 21:19:48 +0000 UTC - event for csi-hostpath-provisioner-0: {kubelet k8s-agentpool-24944505-vmss000001} Started: Started container csi-provisioner Nov 13 21:29:04.833: INFO: At 2019-11-13 21:19:48 +0000 UTC - event for csi-hostpath-resizer-0: {kubelet k8s-agentpool-24944505-vmss000001} Pulled: Successfully pulled image "quay.io/k8scsi/csi-resizer:v0.2.0" Nov 13 21:29:04.833: INFO: At 2019-11-13 21:19:48 +0000 UTC - event for csi-snapshotter-0: {kubelet k8s-agentpool-24944505-vmss000001} Pulling: Pulling image "quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1" Nov 13 21:29:04.833: INFO: At 2019-11-13 21:19:52 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000001} Pulling: Pulling image "quay.io/k8scsi/hostpathplugin:v1.2.0-rc5" Nov 13 21:29:04.833: INFO: At 2019-11-13 21:19:52 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000001} Started: Started container node-driver-registrar Nov 13 21:29:04.833: INFO: At 2019-11-13 21:19:52 +0000 UTC - event for csi-snapshotter-0: {kubelet k8s-agentpool-24944505-vmss000001} Pulled: Successfully pulled image "quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1" Nov 13 21:29:04.833: INFO: At 2019-11-13 21:19:53 +0000 UTC - event for csi-hostpath-resizer-0: {kubelet k8s-agentpool-24944505-vmss000001} Created: Created container csi-resizer Nov 13 21:29:04.833: INFO: At 2019-11-13 21:19:54 +0000 UTC - event for csi-hostpath-attacher-0: {kubelet k8s-agentpool-24944505-vmss000001} Started: Started container csi-attacher Nov 13 21:29:04.833: INFO: At 2019-11-13 21:19:54 +0000 UTC - event for csi-snapshotter-0: {kubelet k8s-agentpool-24944505-vmss000001} Created: Created container csi-snapshotter Nov 13 21:29:04.833: INFO: At 2019-11-13 21:19:57 +0000 UTC - event for csi-hostpath-resizer-0: {kubelet k8s-agentpool-24944505-vmss000001} Started: Started container csi-resizer Nov 13 21:29:04.833: INFO: At 2019-11-13 21:19:59 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000001} Pulled: Successfully pulled image "quay.io/k8scsi/hostpathplugin:v1.2.0-rc5" Nov 13 21:29:04.833: INFO: At 2019-11-13 21:20:00 +0000 UTC - event for csi-snapshotter-0: {kubelet k8s-agentpool-24944505-vmss000001} Started: Started container csi-snapshotter Nov 13 21:29:04.833: INFO: At 2019-11-13 21:20:07 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000001} Created: Created container hostpath Nov 13 21:29:04.833: INFO: At 2019-11-13 21:20:11 +0000 UTC - event for csi-hostpatht6bxt: {csi-hostpath-provisioning-6430_csi-hostpath-provisioner-0_ed7ed059-15c2-417c-b110-3db163b72c5a } Provisioning: External provisioner is provisioning volume for claim "provisioning-6430/csi-hostpatht6bxt" Nov 13 21:29:04.833: INFO: At 2019-11-13 21:20:12 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000001} Pulling: Pulling image "quay.io/k8scsi/livenessprobe:v1.1.0" Nov 13 21:29:04.833: INFO: At 2019-11-13 21:20:12 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000001} Started: Started container hostpath Nov 13 21:29:04.833: INFO: At 2019-11-13 21:20:12 +0000 UTC - event for pod-subpath-test-csi-hostpath-dynamicpv-2c99: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-d596cfa0-6aec-4c5f-8c41-85eada654887" Nov 13 21:29:04.833: INFO: At 2019-11-13 21:20:24 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000001} Pulled: Successfully pulled image "quay.io/k8scsi/livenessprobe:v1.1.0" Nov 13 21:29:04.833: INFO: At 2019-11-13 21:20:30 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000001} Created: Created container liveness-probe Nov 13 21:29:04.833: INFO: At 2019-11-13 21:20:37 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000001} Started: Started container liveness-probe Nov 13 21:29:04.833: INFO: At 2019-11-13 21:20:51 +0000 UTC - event for pod-subpath-test-csi-hostpath-dynamicpv-2c99: {kubelet k8s-agentpool-24944505-vmss000001} Pulling: Pulling image "docker.io/library/busybox:1.29" Nov 13 21:29:04.833: INFO: At 2019-11-13 21:20:57 +0000 UTC - event for pod-subpath-test-csi-hostpath-dynamicpv-2c99: {kubelet k8s-agentpool-24944505-vmss000001} Pulled: Successfully pulled image "docker.io/library/busybox:1.29" Nov 13 21:29:04.833: INFO: At 2019-11-13 21:21:02 +0000 UTC - event for pod-subpath-test-csi-hostpath-dynamicpv-2c99: {kubelet k8s-agentpool-24944505-vmss000001} Created: Created container init-volume-csi-hostpath-dynamicpv-2c99 Nov 13 21:29:04.833: INFO: At 2019-11-13 21:21:07 +0000 UTC - event for pod-subpath-test-csi-hostpath-dynamicpv-2c99: {kubelet k8s-agentpool-24944505-vmss000001} Started: Started container init-volume-csi-hostpath-dynamicpv-2c99 Nov 13 21:29:04.833: INFO: At 2019-11-13 21:22:56 +0000 UTC - event for pod-subpath-test-csi-hostpath-dynamicpv-2c99: {kubelet k8s-agentpool-24944505-vmss000001} Pulling: Pulling image "gcr.io/kubernetes-e2e-test-images/mounttest:1.0" Nov 13 21:29:04.833: INFO: At 2019-11-13 21:22:56 +0000 UTC - event for pod-subpath-test-csi-hostpath-dynamicpv-2c99: {kubelet k8s-agentpool-24944505-vmss000001} Pulled: Successfully pulled image "gcr.io/kubernetes-e2e-test-images/mounttest:1.0" Nov 13 21:29:04.833: INFO: At 2019-11-13 21:22:59 +0000 UTC - event for pod-subpath-test-csi-hostpath-dynamicpv-2c99: {kubelet k8s-agentpool-24944505-vmss000001} Created: Created container test-init-subpath-csi-hostpath-dynamicpv-2c99 Nov 13 21:29:04.833: INFO: At 2019-11-13 21:23:01 +0000 UTC - event for pod-subpath-test-csi-hostpath-dynamicpv-2c99: {kubelet k8s-agentpool-24944505-vmss000001} Started: Started container test-init-subpath-csi-hostpath-dynamicpv-2c99 Nov 13 21:29:04.833: INFO: At 2019-11-13 21:27:07 +0000 UTC - event for csi-hostpath-attacher-0: {taint-controller } TaintManagerEviction: Cancelling deletion of Pod provisioning-6430/csi-hostpath-attacher-0 Nov 13 21:29:04.833: INFO: At 2019-11-13 21:27:07 +0000 UTC - event for csi-hostpath-provisioner-0: {taint-controller } TaintManagerEviction: Cancelling deletion of Pod provisioning-6430/csi-hostpath-provisioner-0 Nov 13 21:29:04.833: INFO: At 2019-11-13 21:27:07 +0000 UTC - event for csi-hostpath-resizer-0: {taint-controller } TaintManagerEviction: Cancelling deletion of Pod provisioning-6430/csi-hostpath-resizer-0 Nov 13 21:29:04.833: INFO: At 2019-11-13 21:27:07 +0000 UTC - event for csi-hostpathplugin-0: {taint-controller } TaintManagerEviction: Cancelling deletion of Pod provisioning-6430/csi-hostpathplugin-0 Nov 13 21:29:04.833: INFO: At 2019-11-13 21:27:07 +0000 UTC - event for csi-snapshotter-0: {taint-controller } TaintManagerEviction: Cancelling deletion of Pod provisioning-6430/csi-snapshotter-0 Nov 13 21:29:04.833: INFO: At 2019-11-13 21:27:07 +0000 UTC - event for pod-subpath-test-csi-hostpath-dynamicpv-2c99: {taint-controller } TaintManagerEviction: Cancelling deletion of Pod provisioning-6430/pod-subpath-test-csi-hostpath-dynamicpv-2c99 Nov 13 21:29:04.888: INFO: POD NODE PHASE GRACE CONDITIONS Nov 13 21:29:04.888: INFO: csi-hostpath-attacher-0 k8s-agentpool-24944505-vmss000001 Running 30s [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:19:05 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:20:03 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:20:03 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:19:05 +0000 UTC }] Nov 13 21:29:04.888: INFO: csi-hostpath-provisioner-0 k8s-agentpool-24944505-vmss000001 Running 30s [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:19:08 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:20:00 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:20:00 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:19:08 +0000 UTC }] Nov 13 21:29:04.888: INFO: csi-hostpath-resizer-0 k8s-agentpool-24944505-vmss000001 Running 30s [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:19:09 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:20:01 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:20:01 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:19:09 +0000 UTC }] Nov 13 21:29:04.888: INFO: csi-hostpathplugin-0 k8s-agentpool-24944505-vmss000001 Running 30s [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:19:07 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:22:51 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:22:51 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:19:07 +0000 UTC }] Nov 13 21:29:04.888: INFO: csi-snapshotter-0 k8s-agentpool-24944505-vmss000001 Running 30s [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:19:10 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:20:03 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:20:03 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:19:10 +0000 UTC }] Nov 13 21:29:04.888: INFO: Nov 13 21:29:04.994: INFO: Logging node info for node k8s-agentpool-24944505-vmss000000 Nov 13 21:29:05.046: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool-24944505-vmss000000 /api/v1/nodes/k8s-agentpool-24944505-vmss000000 5bc7cf9e-80da-4e26-a2c2-20757e870d68 59696 0 2019-11-13 20:42:13 +0000 UTC <nil> <nil> map[agentpool:agentpool beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool-24944505-vmss000000 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: storageprofile:managed storagetier:Premium_LRS] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-1137":"k8s-agentpool-24944505-vmss000000","csi-hostpath-ephemeral-9135":"k8s-agentpool-24944505-vmss000000","csi-hostpath-provisioning-2585":"k8s-agentpool-24944505-vmss000000","csi-hostpath-provisioning-3303":"k8s-agentpool-24944505-vmss000000","csi-hostpath-provisioning-7066":"k8s-agentpool-24944505-vmss000000","csi-hostpath-provisioning-716":"k8s-agentpool-24944505-vmss000000","csi-hostpath-provisioning-833":"k8s-agentpool-24944505-vmss000000","csi-hostpath-volume-9286":"k8s-agentpool-24944505-vmss000000","csi-hostpath-volume-expand-171":"k8s-agentpool-24944505-vmss000000","csi-hostpath-volume-expand-4114":"k8s-agentpool-24944505-vmss000000","csi-hostpath-volume-expand-4561":"k8s-agentpool-24944505-vmss000000","csi-hostpath-volume-expand-9975":"k8s-agentpool-24944505-vmss000000","csi-mock-csi-mock-volumes-2863":"csi-mock-csi-mock-volumes-2863","csi-mock-csi-mock-volumes-5360":"csi-mock-csi-mock-volumes-5360","csi-mock-csi-mock-volumes-6481":"csi-mock-csi-mock-volumes-6481","csi-mock-csi-mock-volumes-6951":"csi-mock-csi-mock-volumes-6951","csi-mock-csi-mock-volumes-8439":"csi-mock-csi-mock-volumes-8439","csi-mock-csi-mock-volumes-8606":"csi-mock-csi-mock-volumes-8606"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-24944505-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16797569024 0} {<nil>} 16403876Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16011137024 0} {<nil>} 15635876Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 21:28:42 +0000 UTC,LastTransitionTime:2019-11-13 20:42:03 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 21:28:42 +0000 UTC,LastTransitionTime:2019-11-13 20:42:03 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 21:28:42 +0000 UTC,LastTransitionTime:2019-11-13 20:42:03 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 21:28:42 +0000 UTC,LastTransitionTime:2019-11-13 20:42:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.248.0.4,},NodeAddress{Type:Hostname,Address:k8s-agentpool-24944505-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:0a2e8938ebfc440180ed5c0b5a9edbe7,SystemUUID:8075A1F3-D764-7248-9570-A764E51D7FF2,BootID:4f2b3cbe-c61c-4c6c-93ab-18fca0df5bc5,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[quay.io/kubernetes_incubator/nfs-provisioner@sha256:df762117e3c891f2d2ddff46ecb0776ba1f9f3c44cfd7739b0683bcd7a7954a8 quay.io/kubernetes_incubator/nfs-provisioner:v2.2.2],SizeBytes:391772778,},ContainerImage{Names:[gcr.io/google-samples/gb-frontend@sha256:35cb427341429fac3df10ff74600ea73e8ec0754d78f9ce89e0b4f3d70d53ba6 gcr.io/google-samples/gb-frontend:v6],SizeBytes:373099368,},ContainerImage{Names:[k8s.gcr.io/etcd@sha256:12c2c5e5731c3bcd56e6f1c05c0f9198b6f06793fa7fca2fb43aab9622dc4afa k8s.gcr.io/etcd:3.3.15],SizeBytes:246640776,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/volume/nfs@sha256:c2ad734346f608a5f7d69cfded93c4e8094069320657bd372d12ba21dea3ea71 gcr.io/kubernetes-e2e-test-images/volume/nfs:1.0],SizeBytes:225358913,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/jessie-dnsutils@sha256:ad583e33cb284f7ef046673809b146ec4053cda19b54a85d2b180a86169715eb gcr.io/kubernetes-e2e-test-images/jessie-dnsutils:1.0],SizeBytes:195659796,},ContainerImage{Names:[httpd@sha256:addd70e4ee83f3bc9a4c1c7c41e37927ba47faf639312fc936df3afad7926f5a httpd:2.4.39-alpine],SizeBytes:126894770,},ContainerImage{Names:[httpd@sha256:eb8ccf084cf3e80eece1add239effefd171eb39adbc154d33c14260d905d4060 httpd:2.4.38-alpine],SizeBytes:123781643,},ContainerImage{Names:[k8s.gcr.io/kubernetes-dashboard-amd64@sha256:0ae6b69432e78069c5ce2bcde0fe409c5c4d6f0f4d9cd50a17974fea38898747 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1],SizeBytes:121711221,},ContainerImage{Names:[gcr.io/kubernetes-helm/tiller@sha256:f6d8f4ab9ba993b5f5b60a6edafe86352eabe474ffeb84cb6c79b8866dce45d1 gcr.io/kubernetes-helm/tiller:v2.11.0],SizeBytes:71821984,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/sample-apiserver@sha256:1bafcc6fb1aa990b487850adba9cadc020e42d7905aa8a30481182a477ba24b0 gcr.io/kubernetes-e2e-test-images/sample-apiserver:1.10],SizeBytes:61365829,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727 gcr.io/kubernetes-e2e-test-images/agnhost:2.6],SizeBytes:57345321,},ContainerImage{Names:[quay.io/k8scsi/csi-provisioner@sha256:0efcb424f1dde9b9fb11a1a14f2e48ab47e1c3f08bc3a929990dcfcb1f7ab34f quay.io/k8scsi/csi-provisioner:v1.4.0-rc1],SizeBytes:54431016,},ContainerImage{Names:[quay.io/k8scsi/csi-snapshotter@sha256:e3d3e742e32d00488fdb401045b9b1d033d7ca0ab6e760f77b24750fc95e5f70 quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1],SizeBytes:51703561,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0],SizeBytes:46226754,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:0aba670b4d9d6b2e720bbf575d733156c676b693ca26501235444490300db838 quay.io/k8scsi/csi-attacher:v1.1.0],SizeBytes:42839085,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:7d46fb6eb8b890dc546029d1565d502b4a1d974d33625c6ee2bc7991b77fc1a1 quay.io/k8scsi/csi-resizer:v0.2.0],SizeBytes:42817100,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:f315c9042e56def3c05c6b04fe79ec9da6d39ddc557ca365a76cf35964ea08b6 quay.io/k8scsi/csi-resizer:v0.1.0],SizeBytes:42623056,},ContainerImage{Names:[k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1],SizeBytes:42541759,},ContainerImage{Names:[gcr.io/google-containers/debian-base@sha256:6966a0aedd7592c18ff2dd803c08bd85780ee19f5e3a2e7cf908a4cd837afcde gcr.io/google-containers/debian-base:0.4.1],SizeBytes:42323657,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nonroot@sha256:d4ede5c74517090b6686219059118ed178cf4620f5db8781b32f806bb1e7395b gcr.io/kubernetes-e2e-test-images/nonroot:1.0],SizeBytes:42321438,},ContainerImage{Names:[redis@sha256:50899ea1ceed33fa03232f3ac57578a424faa1742c1ac9c7a7bdb95cdf19b858 redis:5.0.5-alpine],SizeBytes:29331594,},ContainerImage{Names:[quay.io/k8scsi/hostpathplugin@sha256:b4826e492fc1762fceaf9726f41575ca0a4567864d3d235da874818de18039de quay.io/k8scsi/hostpathplugin:v1.2.0-rc5],SizeBytes:28761497,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7],SizeBytes:22909487,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/echoserver@sha256:e9ba514b896cdf559eef8788b66c2c3ee55f3572df617647b4b0d8b6bf81cf19 gcr.io/kubernetes-e2e-test-images/echoserver:2.2],SizeBytes:21692741,},ContainerImage{Names:[quay.io/k8scsi/mock-driver@sha256:e0eed916b7d970bad2b7d9875f9ad16932f987f0f3d91ec5d86da68b0b5cc9d1 quay.io/k8scsi/mock-driver:v2.1.0],SizeBytes:16226335,},ContainerImage{Names:[nginx@sha256:485b610fefec7ff6c463ced9623314a04ed67e3945b9c08d7e53a47f6d108dc7 nginx:1.14-alpine],SizeBytes:16032814,},ContainerImage{Names:[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0],SizeBytes:15815995,},ContainerImage{Names:[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0],SizeBytes:14967303,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/dnsutils@sha256:2abeee84efb79c14d731966e034af33bf324d3b26ca28497555511ff094b3ddd gcr.io/kubernetes-e2e-test-images/dnsutils:1.1],SizeBytes:9349974,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nonewprivs@sha256:10066e9039219449fe3c81f38fe01928f87914150768ab81b62a468e51fa7411 gcr.io/kubernetes-e2e-test-images/nonewprivs:1.0],SizeBytes:6757579,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nautilus@sha256:33a732d4c42a266912a5091598a0f07653c9134db4b8d571690d8afd509e0bfc gcr.io/kubernetes-e2e-test-images/nautilus:1.0],SizeBytes:4753501,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/kitten@sha256:bcbc4875c982ab39aa7c4f6acf4a287f604e996d9f34a3fbda8c3d1a7457d1f6 gcr.io/kubernetes-e2e-test-images/kitten:1.0],SizeBytes:4747037,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/test-webserver@sha256:7f93d6e32798ff28bc6289254d0c2867fe2c849c8e46edc50f8624734309812e gcr.io/kubernetes-e2e-test-images/test-webserver:1.0],SizeBytes:4732240,},ContainerImage{Names:[alpine@sha256:8421d9a84432575381bfabd248f1eb56f3aa21d9d7cd2511583c68c9b7511d10 alpine:3.7],SizeBytes:4206494,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 gcr.io/kubernetes-e2e-test-images/mounttest:1.0],SizeBytes:1563521,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest-user@sha256:17319ca525ee003681fccf7e8c6b1b910ff4f49b653d939ac7f9b6e7c463933d gcr.io/kubernetes-e2e-test-images/mounttest-user:1.0],SizeBytes:1450451,},ContainerImage{Names:[busybox@sha256:1303dbf110c57f3edf68d9f5a16c082ec06c4cf7604831669faf2c712260b5a0 busybox:latest],SizeBytes:1219790,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29],SizeBytes:1154361,},ContainerImage{Names:[busybox@sha256:bbc3a03235220b170ba48a157dd097dd1379299370e1ed99ce976df0355d24f0 busybox:1.27],SizeBytes:1129289,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea k8s.gcr.io/pause-amd64:3.1 k8s.gcr.io/pause:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 21:29:05.046: INFO: Logging kubelet events for node k8s-agentpool-24944505-vmss000000 Nov 13 21:29:05.102: INFO: Logging pods the kubelet thinks is on node k8s-agentpool-24944505-vmss000000 Nov 13 21:29:05.163: INFO: keyvault-flexvolume-sn47m started at 2019-11-13 20:42:59 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:05.163: INFO: Container keyvault-flexvolume ready: true, restart count 0 Nov 13 21:29:05.163: INFO: blobfuse-flexvol-installer-drhkj started at 2019-11-13 20:42:59 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:05.163: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 Nov 13 21:29:05.163: INFO: kube-proxy-h9n4c started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:05.163: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 21:29:05.163: INFO: kubernetes-dashboard-65966766b9-lq4lm started at 2019-11-13 20:42:59 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:05.163: INFO: Container kubernetes-dashboard ready: true, restart count 0 Nov 13 21:29:05.163: INFO: metrics-server-58ff8c5ddf-r2d7f started at 2019-11-13 20:42:59 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:05.163: INFO: Container metrics-server ready: true, restart count 0 Nov 13 21:29:05.163: INFO: azure-ip-masq-agent-qksp6 started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:05.163: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 21:29:05.163: INFO: tiller-deploy-7559b6b885-qxwkm started at 2019-11-13 20:42:59 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:05.163: INFO: Container tiller ready: true, restart count 0 Nov 13 21:29:05.163: INFO: external-provisioner-fs2p2 started at 2019-11-13 21:19:00 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:05.163: INFO: Container nfs-provisioner ready: true, restart count 0 W1113 21:29:05.215917 92629 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 21:29:05.348: INFO: Latency metrics for node k8s-agentpool-24944505-vmss000000 Nov 13 21:29:05.348: INFO: Logging node info for node k8s-agentpool-24944505-vmss000001 Nov 13 21:29:05.400: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool-24944505-vmss000001 /api/v1/nodes/k8s-agentpool-24944505-vmss000001 da675d2e-0aad-4a8f-b94d-1311fb000563 59487 0 2019-11-13 20:42:24 +0000 UTC <nil> <nil> map[agentpool:agentpool beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool-24944505-vmss000001 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: storageprofile:managed storagetier:Premium_LRS] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-351":"k8s-agentpool-24944505-vmss000001","csi-hostpath-ephemeral-4187":"k8s-agentpool-24944505-vmss000001","csi-hostpath-ephemeral-5584":"k8s-agentpool-24944505-vmss000001","csi-hostpath-provisioning-1638":"k8s-agentpool-24944505-vmss000001","csi-hostpath-provisioning-225":"k8s-agentpool-24944505-vmss000001","csi-hostpath-provisioning-6430":"k8s-agentpool-24944505-vmss000001","csi-hostpath-provisioning-7099":"k8s-agentpool-24944505-vmss000001","csi-hostpath-volume-6291":"k8s-agentpool-24944505-vmss000001","csi-hostpath-volume-expand-7524":"k8s-agentpool-24944505-vmss000001","csi-hostpath-volume-expand-8458":"k8s-agentpool-24944505-vmss000001","csi-mock-csi-mock-volumes-2938":"csi-mock-csi-mock-volumes-2938","csi-mock-csi-mock-volumes-4469":"csi-mock-csi-mock-volumes-4469","csi-mock-csi-mock-volumes-4959":"csi-mock-csi-mock-volumes-4959","csi-mock-csi-mock-volumes-597":"csi-mock-csi-mock-volumes-597","csi-mock-csi-mock-volumes-8812":"csi-mock-csi-mock-volumes-8812","csi-mock-csi-mock-volumes-9088":"csi-mock-csi-mock-volumes-9088","csi-mock-csi-mock-volumes-9179":"csi-mock-csi-mock-volumes-9179"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-24944505-vmss/virtualMachines/1,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16797569024 0} {<nil>} 16403876Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16011137024 0} {<nil>} 15635876Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 21:28:05 +0000 UTC,LastTransitionTime:2019-11-13 20:42:24 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 21:28:05 +0000 UTC,LastTransitionTime:2019-11-13 20:42:24 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 21:28:05 +0000 UTC,LastTransitionTime:2019-11-13 20:42:24 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 21:28:05 +0000 UTC,LastTransitionTime:2019-11-13 21:27:05 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.248.0.5,},NodeAddress{Type:Hostname,Address:k8s-agentpool-24944505-vmss000001,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:6ed41b2c4b884fccaf94748594531e06,SystemUUID:9CE5C2E3-7C34-C64A-9734-2477E19937B7,BootID:306ac3f6-a23b-423e-83c9-e3d5f3938f6c,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[quay.io/kubernetes_incubator/nfs-provisioner@sha256:df762117e3c891f2d2ddff46ecb0776ba1f9f3c44cfd7739b0683bcd7a7954a8 quay.io/kubernetes_incubator/nfs-provisioner:v2.2.2],SizeBytes:391772778,},ContainerImage{Names:[gluster/glusterdynamic-provisioner@sha256:90067cb05a7d217651e84576935934fd9dcff8e6bcdcbaa416bbf36fcd09dbd1 gluster/glusterdynamic-provisioner:v1.0],SizeBytes:373281573,},ContainerImage{Names:[gcr.io/google-samples/gb-frontend@sha256:35cb427341429fac3df10ff74600ea73e8ec0754d78f9ce89e0b4f3d70d53ba6 gcr.io/google-samples/gb-frontend:v6],SizeBytes:373099368,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/volume/nfs@sha256:c2ad734346f608a5f7d69cfded93c4e8094069320657bd372d12ba21dea3ea71 gcr.io/kubernetes-e2e-test-images/volume/nfs:1.0],SizeBytes:225358913,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/jessie-dnsutils@sha256:ad583e33cb284f7ef046673809b146ec4053cda19b54a85d2b180a86169715eb gcr.io/kubernetes-e2e-test-images/jessie-dnsutils:1.0],SizeBytes:195659796,},ContainerImage{Names:[httpd@sha256:addd70e4ee83f3bc9a4c1c7c41e37927ba47faf639312fc936df3afad7926f5a httpd:2.4.39-alpine],SizeBytes:126894770,},ContainerImage{Names:[httpd@sha256:eb8ccf084cf3e80eece1add239effefd171eb39adbc154d33c14260d905d4060 httpd:2.4.38-alpine],SizeBytes:123781643,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727 gcr.io/kubernetes-e2e-test-images/agnhost:2.6],SizeBytes:57345321,},ContainerImage{Names:[quay.io/k8scsi/csi-provisioner@sha256:0efcb424f1dde9b9fb11a1a14f2e48ab47e1c3f08bc3a929990dcfcb1f7ab34f quay.io/k8scsi/csi-provisioner:v1.4.0-rc1],SizeBytes:54431016,},ContainerImage{Names:[quay.io/k8scsi/csi-snapshotter@sha256:e3d3e742e32d00488fdb401045b9b1d033d7ca0ab6e760f77b24750fc95e5f70 quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1],SizeBytes:51703561,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:26fccd7a99d973845df1193b46ebdcc6ab8dc5f6e6be319750c471fce1742d13 quay.io/k8scsi/csi-attacher:v1.2.0],SizeBytes:46226754,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:0aba670b4d9d6b2e720bbf575d733156c676b693ca26501235444490300db838 quay.io/k8scsi/csi-attacher:v1.1.0],SizeBytes:42839085,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:7d46fb6eb8b890dc546029d1565d502b4a1d974d33625c6ee2bc7991b77fc1a1 quay.io/k8scsi/csi-resizer:v0.2.0],SizeBytes:42817100,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:f315c9042e56def3c05c6b04fe79ec9da6d39ddc557ca365a76cf35964ea08b6 quay.io/k8scsi/csi-resizer:v0.1.0],SizeBytes:42623056,},ContainerImage{Names:[k8s.gcr.io/coredns@sha256:e83beb5e43f8513fa735e77ffc5859640baea30a882a11cc75c4c3244a737d3c k8s.gcr.io/coredns:1.5.0],SizeBytes:42488424,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nonroot@sha256:d4ede5c74517090b6686219059118ed178cf4620f5db8781b32f806bb1e7395b gcr.io/kubernetes-e2e-test-images/nonroot:1.0],SizeBytes:42321438,},ContainerImage{Names:[redis@sha256:50899ea1ceed33fa03232f3ac57578a424faa1742c1ac9c7a7bdb95cdf19b858 redis:5.0.5-alpine],SizeBytes:29331594,},ContainerImage{Names:[quay.io/k8scsi/hostpathplugin@sha256:b4826e492fc1762fceaf9726f41575ca0a4567864d3d235da874818de18039de quay.io/k8scsi/hostpathplugin:v1.2.0-rc5],SizeBytes:28761497,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:4fd30d43947d4a54fc89ead7985beecfd3c9b2a93a0655a373b1608ab90bd5af mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.7],SizeBytes:22909487,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/echoserver@sha256:e9ba514b896cdf559eef8788b66c2c3ee55f3572df617647b4b0d8b6bf81cf19 gcr.io/kubernetes-e2e-test-images/echoserver:2.2],SizeBytes:21692741,},ContainerImage{Names:[quay.io/k8scsi/mock-driver@sha256:e0eed916b7d970bad2b7d9875f9ad16932f987f0f3d91ec5d86da68b0b5cc9d1 quay.io/k8scsi/mock-driver:v2.1.0],SizeBytes:16226335,},ContainerImage{Names:[nginx@sha256:485b610fefec7ff6c463ced9623314a04ed67e3945b9c08d7e53a47f6d108dc7 nginx:1.14-alpine],SizeBytes:16032814,},ContainerImage{Names:[quay.io/k8scsi/csi-node-driver-registrar@sha256:13daf82fb99e951a4bff8ae5fc7c17c3a8fe7130be6400990d8f6076c32d4599 quay.io/k8scsi/csi-node-driver-registrar:v1.1.0],SizeBytes:15815995,},ContainerImage{Names:[quay.io/k8scsi/livenessprobe@sha256:dde617756e0f602adc566ab71fd885f1dad451ad3fb063ac991c95a2ff47aea5 quay.io/k8scsi/livenessprobe:v1.1.0],SizeBytes:14967303,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/dnsutils@sha256:2abeee84efb79c14d731966e034af33bf324d3b26ca28497555511ff094b3ddd gcr.io/kubernetes-e2e-test-images/dnsutils:1.1],SizeBytes:9349974,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/nautilus@sha256:33a732d4c42a266912a5091598a0f07653c9134db4b8d571690d8afd509e0bfc gcr.io/kubernetes-e2e-test-images/nautilus:1.0],SizeBytes:4753501,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/kitten@sha256:bcbc4875c982ab39aa7c4f6acf4a287f604e996d9f34a3fbda8c3d1a7457d1f6 gcr.io/kubernetes-e2e-test-images/kitten:1.0],SizeBytes:4747037,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/test-webserver@sha256:7f93d6e32798ff28bc6289254d0c2867fe2c849c8e46edc50f8624734309812e gcr.io/kubernetes-e2e-test-images/test-webserver:1.0],SizeBytes:4732240,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:4206620,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 gcr.io/kubernetes-e2e-test-images/mounttest:1.0],SizeBytes:1563521,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest-user@sha256:17319ca525ee003681fccf7e8c6b1b910ff4f49b653d939ac7f9b6e7c463933d gcr.io/kubernetes-e2e-test-images/mounttest-user:1.0],SizeBytes:1450451,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29],SizeBytes:1154361,},ContainerImage{Names:[busybox@sha256:bbc3a03235220b170ba48a157dd097dd1379299370e1ed99ce976df0355d24f0 busybox:1.27],SizeBytes:1129289,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea k8s.gcr.io/pause-amd64:3.1 k8s.gcr.io/pause:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 21:29:05.401: INFO: Logging kubelet events for node k8s-agentpool-24944505-vmss000001 Nov 13 21:29:05.455: INFO: Logging pods the kubelet thinks is on node k8s-agentpool-24944505-vmss000001 Nov 13 21:29:05.513: INFO: coredns-87f5d796-jd8bc started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:05.513: INFO: Container coredns ready: true, restart count 0 Nov 13 21:29:05.513: INFO: busybox-host-aliases7e358483-64e0-4e66-8435-049f1149ff6a started at 2019-11-13 21:19:26 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:05.513: INFO: Container busybox-host-aliases7e358483-64e0-4e66-8435-049f1149ff6a ready: true, restart count 0 Nov 13 21:29:05.513: INFO: keyvault-flexvolume-7phpr started at 2019-11-13 20:42:58 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:05.513: INFO: Container keyvault-flexvolume ready: true, restart count 0 Nov 13 21:29:05.513: INFO: csi-hostpath-provisioner-0 started at 2019-11-13 21:19:08 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:05.513: INFO: Container csi-provisioner ready: true, restart count 0 Nov 13 21:29:05.513: INFO: external-provisioner-qms6n started at 2019-11-13 21:19:56 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:05.514: INFO: Container nfs-provisioner ready: true, restart count 0 Nov 13 21:29:05.514: INFO: blobfuse-flexvol-installer-cpm6h started at 2019-11-13 20:42:58 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:05.514: INFO: Container blobfuse-flexvol-installer ready: true, restart count 0 Nov 13 21:29:05.514: INFO: csi-hostpathplugin-0 started at 2019-11-13 21:19:07 +0000 UTC (0+3 container statuses recorded) Nov 13 21:29:05.514: INFO: Container hostpath ready: true, restart count 0 Nov 13 21:29:05.514: INFO: Container liveness-probe ready: true, restart count 0 Nov 13 21:29:05.514: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 13 21:29:05.514: INFO: azure-ip-masq-agent-wwpg7 started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:05.514: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 21:29:05.514: INFO: kube-proxy-7bpf7 started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:05.514: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 21:29:05.514: INFO: external-provisioner-rqq6k started at 2019-11-13 21:19:21 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:05.514: INFO: Container nfs-provisioner ready: true, restart count 0 Nov 13 21:29:05.514: INFO: csi-hostpath-resizer-0 started at 2019-11-13 21:19:09 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:05.514: INFO: Container csi-resizer ready: true, restart count 0 Nov 13 21:29:05.514: INFO: recycler-for-nfs-kbrgv started at 2019-11-13 21:19:57 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:05.514: INFO: Container pv-recycler ready: false, restart count 0 Nov 13 21:29:05.514: INFO: affinity-nodeport-transition-24rzm started at 2019-11-13 21:19:04 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:05.514: INFO: Container affinity-nodeport-transition ready: true, restart count 0 Nov 13 21:29:05.514: INFO: csi-hostpath-attacher-0 started at 2019-11-13 21:19:05 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:05.514: INFO: Container csi-attacher ready: true, restart count 0 Nov 13 21:29:05.514: INFO: pfpod started at 2019-11-13 21:20:18 +0000 UTC (0+2 container statuses recorded) Nov 13 21:29:05.514: INFO: Container portforwardtester ready: false, restart count 0 Nov 13 21:29:05.514: INFO: Container readiness ready: false, restart count 0 Nov 13 21:29:05.514: INFO: pod-projected-secrets-2fc08520-7bb7-4fea-8bde-adf3354ecd89 started at 2019-11-13 21:19:00 +0000 UTC (0+3 container statuses recorded) Nov 13 21:29:05.514: INFO: Container creates-volume-test ready: false, restart count 0 Nov 13 21:29:05.514: INFO: Container dels-volume-test ready: false, restart count 0 Nov 13 21:29:05.514: INFO: Container upds-volume-test ready: false, restart count 0 Nov 13 21:29:05.514: INFO: privileged-pod started at 2019-11-13 21:20:06 +0000 UTC (0+2 container statuses recorded) Nov 13 21:29:05.514: INFO: Container not-privileged-container ready: true, restart count 0 Nov 13 21:29:05.514: INFO: Container privileged-container ready: true, restart count 0 Nov 13 21:29:05.514: INFO: csi-snapshotter-0 started at 2019-11-13 21:19:10 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:05.514: INFO: Container csi-snapshotter ready: true, restart count 0 W1113 21:29:05.567496 92629 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 21:29:05.705: INFO: Latency metrics for node k8s-agentpool-24944505-vmss000001 Nov 13 21:29:05.705: INFO: Logging node info for node k8s-master-24944505-vmss000000 Nov 13 21:29:05.757: INFO: Node Info: &Node{ObjectMeta:{k8s-master-24944505-vmss000000 /api/v1/nodes/k8s-master-24944505-vmss000000 048fbae2-b97f-43a2-b471-398180136f12 59528 0 2019-11-13 20:42:13 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-24944505-vmss000000 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-24944505-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284887552 0} {<nil>} 7114148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6498455552 0} {<nil>} 6346148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 21:28:10 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 21:28:10 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 21:28:10 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 21:28:10 +0000 UTC,LastTransitionTime:2019-11-13 20:42:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.4,},NodeAddress{Type:Hostname,Address:k8s-master-24944505-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:7bd754c0dc8647518c3043bcae3a76cc,SystemUUID:FA5CFE99-8148-6C46-B4E6-5EA2A5DE9D2A,BootID:74f7b0a2-e85c-438e-a642-c5b32900bb28,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:3237a53b135d8692ade82eca845fd0259ecbfcbe05fad35400e7d5c2018c4abd k8sprow.azurecr.io/azure-cloud-controller-manager:1194710255657816064-e688ecb0],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 21:29:05.758: INFO: Logging kubelet events for node k8s-master-24944505-vmss000000 Nov 13 21:29:05.823: INFO: Logging pods the kubelet thinks is on node k8s-master-24944505-vmss000000 Nov 13 21:29:05.878: INFO: azure-ip-masq-agent-6pnst started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:05.878: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 21:29:05.878: INFO: kube-proxy-d9jjh started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:05.878: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 21:29:05.878: INFO: cloud-controller-manager-k8s-master-24944505-vmss000000 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:05.878: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 13 21:29:05.878: INFO: kube-addon-manager-k8s-master-24944505-vmss000000 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:05.878: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 13 21:29:05.878: INFO: kube-apiserver-k8s-master-24944505-vmss000000 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:05.878: INFO: Container kube-apiserver ready: true, restart count 0 Nov 13 21:29:05.878: INFO: kube-controller-manager-k8s-master-24944505-vmss000000 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:05.878: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 13 21:29:05.878: INFO: kube-scheduler-k8s-master-24944505-vmss000000 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:05.878: INFO: Container kube-scheduler ready: true, restart count 0 W1113 21:29:05.931258 92629 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 21:29:06.058: INFO: Latency metrics for node k8s-master-24944505-vmss000000 Nov 13 21:29:06.058: INFO: Logging node info for node k8s-master-24944505-vmss000001 Nov 13 21:29:06.110: INFO: Node Info: &Node{ObjectMeta:{k8s-master-24944505-vmss000001 /api/v1/nodes/k8s-master-24944505-vmss000001 a2619878-b119-44f9-b89f-4073bebbb56c 59496 0 2019-11-13 20:42:40 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-24944505-vmss000001 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.6.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-24944505-vmss/virtualMachines/1,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.6.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284883456 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6498451456 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:43:08 +0000 UTC,LastTransitionTime:2019-11-13 20:43:08 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 21:28:06 +0000 UTC,LastTransitionTime:2019-11-13 20:42:34 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 21:28:06 +0000 UTC,LastTransitionTime:2019-11-13 20:42:34 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 21:28:06 +0000 UTC,LastTransitionTime:2019-11-13 20:42:34 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 21:28:06 +0000 UTC,LastTransitionTime:2019-11-13 20:42:40 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.5,},NodeAddress{Type:Hostname,Address:k8s-master-24944505-vmss000001,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:c866a7c1ed64481990416e24387dd317,SystemUUID:EBE64557-2AC8-5740-9FFE-AB9F12412FBC,BootID:3dbb0e0e-0479-4821-bc3f-a8269d250e4c,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:3237a53b135d8692ade82eca845fd0259ecbfcbe05fad35400e7d5c2018c4abd k8sprow.azurecr.io/azure-cloud-controller-manager:1194710255657816064-e688ecb0],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 21:29:06.110: INFO: Logging kubelet events for node k8s-master-24944505-vmss000001 Nov 13 21:29:06.179: INFO: Logging pods the kubelet thinks is on node k8s-master-24944505-vmss000001 Nov 13 21:29:06.233: INFO: kube-addon-manager-k8s-master-24944505-vmss000001 started at 2019-11-13 20:42:28 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:06.233: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 13 21:29:06.233: INFO: kube-apiserver-k8s-master-24944505-vmss000001 started at 2019-11-13 20:42:28 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:06.233: INFO: Container kube-apiserver ready: true, restart count 0 Nov 13 21:29:06.233: INFO: kube-controller-manager-k8s-master-24944505-vmss000001 started at 2019-11-13 20:42:28 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:06.233: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 13 21:29:06.233: INFO: kube-proxy-w4bz5 started at 2019-11-13 20:42:41 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:06.233: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 21:29:06.233: INFO: azure-ip-masq-agent-bs52d started at 2019-11-13 20:42:41 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:06.233: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 21:29:06.233: INFO: kube-scheduler-k8s-master-24944505-vmss000001 started at 2019-11-13 20:42:28 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:06.233: INFO: Container kube-scheduler ready: true, restart count 0 Nov 13 21:29:06.233: INFO: cloud-controller-manager-k8s-master-24944505-vmss000001 started at 2019-11-13 20:42:28 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:06.233: INFO: Container cloud-controller-manager ready: true, restart count 0 W1113 21:29:06.286674 92629 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 21:29:06.404: INFO: Latency metrics for node k8s-master-24944505-vmss000001 Nov 13 21:29:06.404: INFO: Logging node info for node k8s-master-24944505-vmss000002 Nov 13 21:29:06.456: INFO: Node Info: &Node{ObjectMeta:{k8s-master-24944505-vmss000002 /api/v1/nodes/k8s-master-24944505-vmss000002 f3b05267-a009-4dba-9f97-4f772e48707e 59524 0 2019-11-13 20:42:13 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-24944505-vmss000002 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.4.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-24944505-vmss/virtualMachines/2,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284887552 0} {<nil>} 7114148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6498455552 0} {<nil>} 6346148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 21:28:09 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 21:28:09 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 21:28:09 +0000 UTC,LastTransitionTime:2019-11-13 20:42:10 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 21:28:09 +0000 UTC,LastTransitionTime:2019-11-13 20:42:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.6,},NodeAddress{Type:Hostname,Address:k8s-master-24944505-vmss000002,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:fc1777f37ced4b9c97acb9b30e276c4f,SystemUUID:E9DF19DC-D02C-714C-A7D6-0BA6CE959C55,BootID:622958a8-d45d-4edf-9266-4dbb759e2359,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:3237a53b135d8692ade82eca845fd0259ecbfcbe05fad35400e7d5c2018c4abd k8sprow.azurecr.io/azure-cloud-controller-manager:1194710255657816064-e688ecb0],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 21:29:06.456: INFO: Logging kubelet events for node k8s-master-24944505-vmss000002 Nov 13 21:29:06.515: INFO: Logging pods the kubelet thinks is on node k8s-master-24944505-vmss000002 Nov 13 21:29:06.571: INFO: kube-scheduler-k8s-master-24944505-vmss000002 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:06.571: INFO: Container kube-scheduler ready: true, restart count 0 Nov 13 21:29:06.571: INFO: cloud-controller-manager-k8s-master-24944505-vmss000002 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:06.571: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 13 21:29:06.571: INFO: azure-ip-masq-agent-6skf2 started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:06.571: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 21:29:06.571: INFO: kube-proxy-xwz78 started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:06.571: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 21:29:06.571: INFO: kube-addon-manager-k8s-master-24944505-vmss000002 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:06.571: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 13 21:29:06.571: INFO: kube-apiserver-k8s-master-24944505-vmss000002 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:06.571: INFO: Container kube-apiserver ready: true, restart count 0 Nov 13 21:29:06.571: INFO: kube-controller-manager-k8s-master-24944505-vmss000002 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:06.571: INFO: Container kube-controller-manager ready: true, restart count 0 W1113 21:29:06.624785 92629 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 21:29:06.739: INFO: Latency metrics for node k8s-master-24944505-vmss000002 Nov 13 21:29:06.739: INFO: Logging node info for node k8s-master-24944505-vmss000003 Nov 13 21:29:06.790: INFO: Node Info: &Node{ObjectMeta:{k8s-master-24944505-vmss000003 /api/v1/nodes/k8s-master-24944505-vmss000003 888cc19e-c62d-4945-a425-97d14e38a0b8 59804 0 2019-11-13 20:42:35 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-2 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-24944505-vmss000003 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.5.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-24944505-vmss/virtualMachines/3,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.5.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284887552 0} {<nil>} 7114148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6498455552 0} {<nil>} 6346148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 21:29:02 +0000 UTC,LastTransitionTime:2019-11-13 20:42:29 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 21:29:02 +0000 UTC,LastTransitionTime:2019-11-13 20:42:29 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 21:29:02 +0000 UTC,LastTransitionTime:2019-11-13 20:42:29 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 21:29:02 +0000 UTC,LastTransitionTime:2019-11-13 20:42:35 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.7,},NodeAddress{Type:Hostname,Address:k8s-master-24944505-vmss000003,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:2da5fe5a442b4e178a2b62ba91366e4b,SystemUUID:D2D1E816-D70E-3D40-B959-B00A2B76A83A,BootID:c5929aa1-3dac-45d5-bb2c-174c73c7300e,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:3237a53b135d8692ade82eca845fd0259ecbfcbe05fad35400e7d5c2018c4abd k8sprow.azurecr.io/azure-cloud-controller-manager:1194710255657816064-e688ecb0],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 21:29:06.790: INFO: Logging kubelet events for node k8s-master-24944505-vmss000003 Nov 13 21:29:06.846: INFO: Logging pods the kubelet thinks is on node k8s-master-24944505-vmss000003 Nov 13 21:29:06.901: INFO: kube-proxy-kgh5h started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:06.901: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 21:29:06.901: INFO: kube-addon-manager-k8s-master-24944505-vmss000003 started at 2019-11-13 20:42:22 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:06.901: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 13 21:29:06.901: INFO: kube-apiserver-k8s-master-24944505-vmss000003 started at 2019-11-13 20:42:23 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:06.901: INFO: Container kube-apiserver ready: true, restart count 0 Nov 13 21:29:06.901: INFO: kube-controller-manager-k8s-master-24944505-vmss000003 started at 2019-11-13 20:42:23 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:06.901: INFO: Container kube-controller-manager ready: true, restart count 0 Nov 13 21:29:06.901: INFO: kube-scheduler-k8s-master-24944505-vmss000003 started at 2019-11-13 20:42:23 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:06.901: INFO: Container kube-scheduler ready: true, restart count 0 Nov 13 21:29:06.901: INFO: cloud-controller-manager-k8s-master-24944505-vmss000003 started at 2019-11-13 20:42:44 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:06.901: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 13 21:29:06.901: INFO: azure-ip-masq-agent-nlqss started at 2019-11-13 20:42:37 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:06.901: INFO: Container azure-ip-masq-agent ready: true, restart count 0 W1113 21:29:06.953245 92629 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 21:29:07.068: INFO: Latency metrics for node k8s-master-24944505-vmss000003 Nov 13 21:29:07.068: INFO: Logging node info for node k8s-master-24944505-vmss000004 Nov 13 21:29:07.120: INFO: Node Info: &Node{ObjectMeta:{k8s-master-24944505-vmss000004 /api/v1/nodes/k8s-master-24944505-vmss000004 a26d844f-0612-406b-9a65-7e2ac952a2d5 59527 0 2019-11-13 20:42:14 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_DS2_v2 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:master kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-master-24944505-vmss000004 kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/master:] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-master-24944505-vmss/virtualMachines/4,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:true,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.244.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7284887552 0} {<nil>} 7114148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{6498455552 0} {<nil>} 6346148Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 21:28:10 +0000 UTC,LastTransitionTime:2019-11-13 20:42:09 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 21:28:10 +0000 UTC,LastTransitionTime:2019-11-13 20:42:09 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2019-11-13 21:28:10 +0000 UTC,LastTransitionTime:2019-11-13 20:42:09 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2019-11-13 21:28:10 +0000 UTC,LastTransitionTime:2019-11-13 20:42:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.8,},NodeAddress{Type:Hostname,Address:k8s-master-24944505-vmss000004,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:2208959aca1045d9b48fce8d784c1e41,SystemUUID:E253DB3A-3A4D-5648-8169-B2E2976CFB4B,BootID:09bc6101-2da9-4c8e-8274-43db8c5fe1cf,KernelVersion:4.15.0-1063-azure,OSImage:Ubuntu 16.04.6 LTS,ContainerRuntimeVersion:docker://3.0.6,KubeletVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,KubeProxyVersion:v1.16.4-beta.0.1+d70a3ca08fe72a,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8sprow.azurecr.io/hyperkube-amd64@sha256:5c7bd604298b2a1489f21cacb17fb602ad215950d352fb5ebdb8629a4fc41e82 k8sprow.azurecr.io/hyperkube-amd64:azure-e2e-1194710255657816064-345e69eb],SizeBytes:604811790,},ContainerImage{Names:[k8sprow.azurecr.io/azure-cloud-controller-manager@sha256:3237a53b135d8692ade82eca845fd0259ecbfcbe05fad35400e7d5c2018c4abd k8sprow.azurecr.io/azure-cloud-controller-manager:1194710255657816064-e688ecb0],SizeBytes:92595467,},ContainerImage{Names:[k8s.gcr.io/kube-addon-manager-amd64@sha256:382c220b3531d9f95bf316a16b7282cc2ef929cd8a89a9dd3f5933edafc41a8e k8s.gcr.io/kube-addon-manager-amd64:v9.0.1],SizeBytes:83076194,},ContainerImage{Names:[k8s.gcr.io/ip-masq-agent-amd64@sha256:269e0fb9d53fd37f7a135d6a55ea265a67279ba218aa148323f015cf70167340 k8s.gcr.io/ip-masq-agent-amd64:v2.3.0],SizeBytes:50144412,},ContainerImage{Names:[k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 13 21:29:07.121: INFO: Logging kubelet events for node k8s-master-24944505-vmss000004 Nov 13 21:29:07.177: INFO: Logging pods the kubelet thinks is on node k8s-master-24944505-vmss000004 Nov 13 21:29:07.232: INFO: kube-scheduler-k8s-master-24944505-vmss000004 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:07.232: INFO: Container kube-scheduler ready: true, restart count 0 Nov 13 21:29:07.232: INFO: azure-ip-masq-agent-xnw2h started at 2019-11-13 20:42:38 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:07.232: INFO: Container azure-ip-masq-agent ready: true, restart count 0 Nov 13 21:29:07.232: INFO: kube-proxy-fld6p started at 2019-11-13 20:42:38 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:07.232: INFO: Container kube-proxy ready: true, restart count 0 Nov 13 21:29:07.232: INFO: cloud-controller-manager-k8s-master-24944505-vmss000004 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:07.232: INFO: Container cloud-controller-manager ready: true, restart count 0 Nov 13 21:29:07.232: INFO: kube-addon-manager-k8s-master-24944505-vmss000004 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:07.232: INFO: Container kube-addon-manager ready: true, restart count 0 Nov 13 21:29:07.232: INFO: kube-apiserver-k8s-master-24944505-vmss000004 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:07.232: INFO: Container kube-apiserver ready: true, restart count 0 Nov 13 21:29:07.232: INFO: kube-controller-manager-k8s-master-24944505-vmss000004 started at 2019-11-13 20:42:03 +0000 UTC (0+1 container statuses recorded) Nov 13 21:29:07.232: INFO: Container kube-controller-manager ready: true, restart count 0 W1113 21:29:07.285129 92629 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. Nov 13 21:29:07.399: INFO: Latency metrics for node k8s-master-24944505-vmss000004 Nov 13 21:29:07.399: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "provisioning-6430" for this suite. Nov 13 21:39:07.740: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered Nov 13 21:39:08.069: INFO: namespace: provisioning-6430, resource: pods, items remaining: 5 Nov 13 21:39:09.545: INFO: namespace: provisioning-6430, DeletionTimetamp: 2019-11-13 21:29:07 +0000 UTC, Finalizers: [kubernetes], Phase: Terminating Nov 13 21:39:09.597: INFO: namespace: provisioning-6430, total namespaces: 12, active: 8, terminating: 4 Nov 13 21:39:09.650: INFO: POD NODE PHASE GRACE CONDITIONS Nov 13 21:39:09.650: INFO: csi-hostpath-attacher-0 k8s-agentpool-24944505-vmss000001 Running 30s [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:19:05 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:39:00 +0000 UTC ContainersNotReady containers with unready status: [csi-attacher]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:39:00 +0000 UTC ContainersNotReady containers with unready status: [csi-attacher]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:19:05 +0000 UTC }] Nov 13 21:39:09.650: INFO: csi-hostpath-provisioner-0 k8s-agentpool-24944505-vmss000001 Running 30s [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:19:08 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:20:00 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:20:00 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:19:08 +0000 UTC }] Nov 13 21:39:09.650: INFO: csi-hostpath-resizer-0 k8s-agentpool-24944505-vmss000001 Running 30s [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:19:09 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:39:00 +0000 UTC ContainersNotReady containers with unready status: [csi-resizer]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:39:00 +0000 UTC ContainersNotReady containers with unready status: [csi-resizer]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:19:09 +0000 UTC }] Nov 13 21:39:09.650: INFO: csi-hostpathplugin-0 k8s-agentpool-24944505-vmss000001 Running 30s [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:19:07 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:39:00 +0000 UTC ContainersNotReady containers with unready status: [node-driver-registrar hostpath liveness-probe]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:39:00 +0000 UTC ContainersNotReady containers with unready status: [node-driver-registrar hostpath liveness-probe]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:19:07 +0000 UTC }] Nov 13 21:39:09.650: INFO: csi-snapshotter-0 k8s-agentpool-24944505-vmss000001 Running 30s [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:19:10 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:20:03 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:20:03 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 21:19:10 +0000 UTC }] Nov 13 21:39:09.650: INFO: Nov 13 21:39:09.650: INFO: Couldn't delete ns: "provisioning-6430": namespace provisioning-6430 was not deleted with limit: timed out waiting for the condition, pods remaining: 5 (&errors.errorString{s:"namespace provisioning-6430 was not deleted with limit: timed out waiting for the condition, pods remaining: 5"})
Find pod-subpath-test-csi-hostpath-dynamicpv-2c99 mentions in log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[sig\-storage\]\sCSI\sVolumes\s\[Driver\:\scsi\-hostpath\]\s\[Testpattern\:\sDynamic\sPV\s\(default\sfs\)\]\ssubPath\sshould\ssupport\sreadOnly\sdirectory\sspecified\sin\sthe\svolumeMount$'
test/e2e/storage/testsuites/subpath.go:347 Nov 13 20:57:47.376: Unexpected error: <*errors.errorString | 0xc001ce5b20>: { s: "PersistentVolumeClaims [csi-hostpath5cdw6] not all in phase Bound within 5m0s", } PersistentVolumeClaims [csi-hostpath5cdw6] not all in phase Bound within 5m0s occurred test/e2e/storage/testsuites/base.go:366
[BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath test/e2e/storage/testsuites/base.go:93 [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath test/e2e/framework/framework.go:151 �[1mSTEP�[0m: Creating a kubernetes client Nov 13 20:52:43.810: INFO: >>> kubeConfig: /workspace/aks880331199/kubeconfig/kubeconfig.westus2.json �[1mSTEP�[0m: Building a namespace api object, basename provisioning �[1mSTEP�[0m: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in provisioning-5842 �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace [It] should support readOnly directory specified in the volumeMount test/e2e/storage/testsuites/subpath.go:347 �[1mSTEP�[0m: deploying csi-hostpath driver Nov 13 20:52:44.447: INFO: creating *v1.ServiceAccount: provisioning-5842/csi-attacher Nov 13 20:52:44.505: INFO: creating *v1.ClusterRole: external-attacher-runner-provisioning-5842 Nov 13 20:52:44.505: INFO: Define cluster role external-attacher-runner-provisioning-5842 Nov 13 20:52:44.566: INFO: creating *v1.ClusterRoleBinding: csi-attacher-role-provisioning-5842 Nov 13 20:52:44.625: INFO: creating *v1.Role: provisioning-5842/external-attacher-cfg-provisioning-5842 Nov 13 20:52:44.684: INFO: creating *v1.RoleBinding: provisioning-5842/csi-attacher-role-cfg Nov 13 20:52:44.758: INFO: creating *v1.ServiceAccount: provisioning-5842/csi-provisioner Nov 13 20:52:44.825: INFO: creating *v1.ClusterRole: external-provisioner-runner-provisioning-5842 Nov 13 20:52:44.825: INFO: Define cluster role external-provisioner-runner-provisioning-5842 Nov 13 20:52:44.888: INFO: creating *v1.ClusterRoleBinding: csi-provisioner-role-provisioning-5842 Nov 13 20:52:44.948: INFO: creating *v1.Role: provisioning-5842/external-provisioner-cfg-provisioning-5842 Nov 13 20:52:45.007: INFO: creating *v1.RoleBinding: provisioning-5842/csi-provisioner-role-cfg Nov 13 20:52:45.067: INFO: creating *v1.ServiceAccount: provisioning-5842/csi-snapshotter Nov 13 20:52:45.131: INFO: creating *v1.ClusterRole: external-snapshotter-runner-provisioning-5842 Nov 13 20:52:45.131: INFO: Define cluster role external-snapshotter-runner-provisioning-5842 Nov 13 20:52:45.234: INFO: creating *v1.ClusterRoleBinding: csi-snapshotter-role-provisioning-5842 Nov 13 20:52:45.303: INFO: creating *v1.Role: provisioning-5842/external-snapshotter-leaderelection-provisioning-5842 Nov 13 20:52:45.363: INFO: creating *v1.RoleBinding: provisioning-5842/external-snapshotter-leaderelection Nov 13 20:52:45.434: INFO: creating *v1.ServiceAccount: provisioning-5842/csi-resizer Nov 13 20:52:45.494: INFO: creating *v1.ClusterRole: external-resizer-runner-provisioning-5842 Nov 13 20:52:45.494: INFO: Define cluster role external-resizer-runner-provisioning-5842 Nov 13 20:52:45.592: INFO: creating *v1.ClusterRoleBinding: csi-resizer-role-provisioning-5842 Nov 13 20:52:45.658: INFO: creating *v1.Role: provisioning-5842/external-resizer-cfg-provisioning-5842 Nov 13 20:52:45.718: INFO: creating *v1.RoleBinding: provisioning-5842/csi-resizer-role-cfg Nov 13 20:52:45.777: INFO: creating *v1.Service: provisioning-5842/csi-hostpath-attacher Nov 13 20:52:45.857: INFO: creating *v1.StatefulSet: provisioning-5842/csi-hostpath-attacher Nov 13 20:52:45.918: INFO: creating *v1beta1.CSIDriver: csi-hostpath-provisioning-5842 Nov 13 20:52:45.980: INFO: creating *v1.Service: provisioning-5842/csi-hostpathplugin Nov 13 20:52:46.055: INFO: creating *v1.StatefulSet: provisioning-5842/csi-hostpathplugin Nov 13 20:52:46.127: INFO: creating *v1.Service: provisioning-5842/csi-hostpath-provisioner Nov 13 20:52:46.201: INFO: creating *v1.StatefulSet: provisioning-5842/csi-hostpath-provisioner Nov 13 20:52:46.269: INFO: creating *v1.Service: provisioning-5842/csi-hostpath-resizer Nov 13 20:52:46.352: INFO: creating *v1.StatefulSet: provisioning-5842/csi-hostpath-resizer Nov 13 20:52:46.425: INFO: creating *v1.Service: provisioning-5842/csi-snapshotter Nov 13 20:52:46.517: INFO: creating *v1.StatefulSet: provisioning-5842/csi-snapshotter Nov 13 20:52:46.583: INFO: creating *v1.ClusterRoleBinding: psp-csi-hostpath-role-provisioning-5842 Nov 13 20:52:46.645: INFO: Test running for native CSI Driver, not checking metrics Nov 13 20:52:46.645: INFO: Creating resource for dynamic PV �[1mSTEP�[0m: creating a StorageClass provisioning-5842-csi-hostpath-provisioning-5842-scggmfg �[1mSTEP�[0m: creating a claim Nov 13 20:52:46.707: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil Nov 13 20:52:46.776: INFO: Waiting up to 5m0s for PersistentVolumeClaims [csi-hostpath5cdw6] to have phase Bound Nov 13 20:52:46.832: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:52:48.888: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:52:50.944: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:52:53.004: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:52:55.060: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:52:57.115: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:52:59.221: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:53:01.282: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:53:03.338: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:53:05.395: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:53:07.454: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:53:09.512: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:53:11.582: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:53:13.669: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:53:15.749: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:53:17.809: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:53:19.864: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:53:21.920: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:53:23.975: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:53:26.031: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:53:28.093: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:53:30.148: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:53:32.203: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:53:34.262: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:53:36.317: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:53:38.373: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:53:40.429: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:53:42.485: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:53:44.540: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:53:46.596: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:53:48.663: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:53:50.726: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:53:52.784: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:53:54.846: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:53:56.911: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:53:58.967: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:54:01.023: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:54:03.079: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:54:05.143: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:54:07.198: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:54:09.266: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:54:11.322: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:54:13.377: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:54:15.434: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:54:17.489: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:54:19.548: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:54:21.623: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:54:23.679: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:54:25.736: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:54:27.793: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:54:29.848: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:54:31.903: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:54:33.959: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:54:36.015: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:54:38.071: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:54:40.128: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:54:42.184: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:54:44.240: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:54:46.299: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:54:48.355: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:54:50.411: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:54:52.466: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:54:54.532: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:54:56.592: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:54:58.648: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:55:00.704: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:55:02.760: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:55:04.816: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:55:06.874: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:55:08.929: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:55:10.984: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:55:13.043: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:55:15.098: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:55:17.153: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:55:19.218: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:55:21.273: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:55:23.330: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:55:25.386: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:55:27.442: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:55:29.501: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:55:31.573: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:55:33.643: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:55:35.698: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:55:37.763: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:55:39.826: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:55:41.889: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:55:43.960: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:55:46.016: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:55:48.072: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:55:50.130: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:55:52.199: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:55:54.256: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:55:56.312: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:55:58.370: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:56:00.427: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:56:02.483: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:56:04.538: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:56:06.594: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:56:08.650: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:56:10.710: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:56:12.766: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:56:14.840: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:56:16.906: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:56:18.962: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:56:21.024: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:56:23.081: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:56:25.136: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:56:27.192: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:56:29.248: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:56:31.303: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:56:33.361: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:56:35.416: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:56:37.476: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:56:39.532: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:56:41.591: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:56:43.646: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:56:45.702: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:56:47.761: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:56:49.823: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:56:51.880: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:56:53.935: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:56:55.994: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:56:58.050: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:57:00.106: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:57:02.165: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:57:04.221: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:57:06.277: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:57:08.339: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:57:10.395: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:57:12.452: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:57:14.508: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:57:16.564: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:57:18.621: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:57:20.677: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:57:22.734: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:57:24.793: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:57:26.850: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:57:28.907: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:57:30.964: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:57:33.020: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:57:35.078: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:57:37.135: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:57:39.195: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:57:41.259: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:57:43.317: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:57:45.375: INFO: PersistentVolumeClaim csi-hostpath5cdw6 found but phase is Pending instead of Bound. Nov 13 20:57:47.376: FAIL: Unexpected error: <*errors.errorString | 0xc001ce5b20>: { s: "PersistentVolumeClaims [csi-hostpath5cdw6] not all in phase Bound within 5m0s", } PersistentVolumeClaims [csi-hostpath5cdw6] not all in phase Bound within 5m0s occurred [AfterEach] [Testpattern: Dynamic PV (default fs)] subPath test/e2e/framework/framework.go:152 �[1mSTEP�[0m: Collecting events from namespace "provisioning-5842". �[1mSTEP�[0m: Found 53 events. Nov 13 20:57:47.489: INFO: At 2019-11-13 20:52:45 +0000 UTC - event for csi-hostpath-attacher: {statefulset-controller } SuccessfulCreate: create Pod csi-hostpath-attacher-0 in StatefulSet csi-hostpath-attacher successful Nov 13 20:57:47.489: INFO: At 2019-11-13 20:52:46 +0000 UTC - event for csi-hostpath-provisioner: {statefulset-controller } SuccessfulCreate: create Pod csi-hostpath-provisioner-0 in StatefulSet csi-hostpath-provisioner successful Nov 13 20:57:47.489: INFO: At 2019-11-13 20:52:46 +0000 UTC - event for csi-hostpath-resizer: {statefulset-controller } SuccessfulCreate: create Pod csi-hostpath-resizer-0 in StatefulSet csi-hostpath-resizer successful Nov 13 20:57:47.489: INFO: At 2019-11-13 20:52:46 +0000 UTC - event for csi-hostpath5cdw6: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "csi-hostpath-provisioning-5842" or manually created by system administrator Nov 13 20:57:47.489: INFO: At 2019-11-13 20:52:46 +0000 UTC - event for csi-hostpathplugin: {statefulset-controller } SuccessfulCreate: create Pod csi-hostpathplugin-0 in StatefulSet csi-hostpathplugin successful Nov 13 20:57:47.489: INFO: At 2019-11-13 20:52:46 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:57:47.489: INFO: At 2019-11-13 20:52:46 +0000 UTC - event for csi-snapshotter: {statefulset-controller } SuccessfulCreate: create Pod csi-snapshotter-0 in StatefulSet csi-snapshotter successful Nov 13 20:57:47.489: INFO: At 2019-11-13 20:52:48 +0000 UTC - event for csi-hostpath-attacher-0: {kubelet k8s-agentpool-24944505-vmss000000} Pulling: Pulling image "quay.io/k8scsi/csi-attacher:v1.2.0" Nov 13 20:57:47.489: INFO: At 2019-11-13 20:52:49 +0000 UTC - event for csi-hostpath-provisioner-0: {kubelet k8s-agentpool-24944505-vmss000000} Pulling: Pulling image "quay.io/k8scsi/csi-provisioner:v1.4.0-rc1" Nov 13 20:57:47.489: INFO: At 2019-11-13 20:52:49 +0000 UTC - event for csi-hostpath-resizer-0: {kubelet k8s-agentpool-24944505-vmss000000} Pulling: Pulling image "quay.io/k8scsi/csi-resizer:v0.2.0" Nov 13 20:57:47.489: INFO: At 2019-11-13 20:52:50 +0000 UTC - event for csi-hostpath-attacher-0: {kubelet k8s-agentpool-24944505-vmss000000} Pulled: Successfully pulled image "quay.io/k8scsi/csi-attacher:v1.2.0" Nov 13 20:57:47.489: INFO: At 2019-11-13 20:52:50 +0000 UTC - event for csi-snapshotter-0: {kubelet k8s-agentpool-24944505-vmss000000} Pulling: Pulling image "quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1" Nov 13 20:57:47.489: INFO: At 2019-11-13 20:52:51 +0000 UTC - event for csi-hostpath-attacher-0: {kubelet k8s-agentpool-24944505-vmss000000} Created: Created container csi-attacher Nov 13 20:57:47.489: INFO: At 2019-11-13 20:52:52 +0000 UTC - event for csi-hostpath-attacher-0: {kubelet k8s-agentpool-24944505-vmss000000} Started: Started container csi-attacher Nov 13 20:57:47.489: INFO: At 2019-11-13 20:52:52 +0000 UTC - event for csi-hostpath-provisioner-0: {kubelet k8s-agentpool-24944505-vmss000000} Created: Created container csi-provisioner Nov 13 20:57:47.489: INFO: At 2019-11-13 20:52:52 +0000 UTC - event for csi-hostpath-provisioner-0: {kubelet k8s-agentpool-24944505-vmss000000} Pulled: Successfully pulled image "quay.io/k8scsi/csi-provisioner:v1.4.0-rc1" Nov 13 20:57:47.489: INFO: At 2019-11-13 20:52:53 +0000 UTC - event for csi-hostpath-provisioner-0: {kubelet k8s-agentpool-24944505-vmss000000} Started: Started container csi-provisioner Nov 13 20:57:47.489: INFO: At 2019-11-13 20:52:53 +0000 UTC - event for csi-hostpath-resizer-0: {kubelet k8s-agentpool-24944505-vmss000000} Pulled: Successfully pulled image "quay.io/k8scsi/csi-resizer:v0.2.0" Nov 13 20:57:47.489: INFO: At 2019-11-13 20:52:54 +0000 UTC - event for csi-hostpath-resizer-0: {kubelet k8s-agentpool-24944505-vmss000000} Created: Created container csi-resizer Nov 13 20:57:47.489: INFO: At 2019-11-13 20:52:54 +0000 UTC - event for csi-hostpath-resizer-0: {kubelet k8s-agentpool-24944505-vmss000000} Started: Started container csi-resizer Nov 13 20:57:47.489: INFO: At 2019-11-13 20:52:54 +0000 UTC - event for csi-snapshotter-0: {kubelet k8s-agentpool-24944505-vmss000000} Pulled: Successfully pulled image "quay.io/k8scsi/csi-snapshotter:v2.0.0-rc1" Nov 13 20:57:47.489: INFO: At 2019-11-13 20:52:55 +0000 UTC - event for csi-snapshotter-0: {kubelet k8s-agentpool-24944505-vmss000000} Started: Started container csi-snapshotter Nov 13 20:57:47.489: INFO: At 2019-11-13 20:52:55 +0000 UTC - event for csi-snapshotter-0: {kubelet k8s-agentpool-24944505-vmss000000} Created: Created container csi-snapshotter Nov 13 20:57:47.489: INFO: At 2019-11-13 20:52:56 +0000 UTC - event for csi-hostpathplugin: {statefulset-controller } FailedCreate: create Pod csi-hostpathplugin-0 in StatefulSet csi-hostpathplugin failed error: The POST operation against Pod could not be completed at this time, please try again. Nov 13 20:57:47.489: INFO: At 2019-11-13 20:52:56 +0000 UTC - event for csi-hostpathplugin: {statefulset-controller } RecreatingFailedPod: StatefulSet provisioning-5842/csi-hostpathplugin is recreating failed Pod csi-hostpathplugin-0 Nov 13 20:57:47.489: INFO: At 2019-11-13 20:52:56 +0000 UTC - event for csi-hostpathplugin: {statefulset-controller } SuccessfulDelete: delete Pod csi-hostpathplugin-0 in StatefulSet csi-hostpathplugin successful Nov 13 20:57:47.490: INFO: At 2019-11-13 20:52:56 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:57:47.490: INFO: At 2019-11-13 20:53:08 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:57:47.490: INFO: At 2019-11-13 20:53:16 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:57:47.490: INFO: At 2019-11-13 20:53:29 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:57:47.490: INFO: At 2019-11-13 20:53:40 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:57:47.490: INFO: At 2019-11-13 20:53:47 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:57:47.490: INFO: At 2019-11-13 20:53:57 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:57:47.490: INFO: At 2019-11-13 20:54:05 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:57:47.490: INFO: At 2019-11-13 20:54:18 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:57:47.490: INFO: At 2019-11-13 20:54:24 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:57:47.490: INFO: At 2019-11-13 20:54:35 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:57:47.490: INFO: At 2019-11-13 20:54:46 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:57:47.490: INFO: At 2019-11-13 20:54:54 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:57:47.490: INFO: At 2019-11-13 20:55:06 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:57:47.490: INFO: At 2019-11-13 20:55:19 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:57:47.490: INFO: At 2019-11-13 20:55:35 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:57:47.490: INFO: At 2019-11-13 20:55:46 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:57:47.490: INFO: At 2019-11-13 20:56:20 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:57:47.490: INFO: At 2019-11-13 20:56:25 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:57:47.490: INFO: At 2019-11-13 20:56:37 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:57:47.490: INFO: At 2019-11-13 20:56:46 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:57:47.490: INFO: At 2019-11-13 20:56:57 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:57:47.490: INFO: At 2019-11-13 20:57:05 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:57:47.490: INFO: At 2019-11-13 20:57:19 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:57:47.490: INFO: At 2019-11-13 20:57:32 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:57:47.490: INFO: At 2019-11-13 20:57:37 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:57:47.490: INFO: At 2019-11-13 20:57:46 +0000 UTC - event for csi-hostpathplugin-0: {kubelet k8s-agentpool-24944505-vmss000000} PodFitsHostPorts: Predicate PodFitsHostPorts failed Nov 13 20:57:47.548: INFO: POD NODE PHASE GRACE CONDITIONS Nov 13 20:57:47.548: INFO: csi-hostpath-attacher-0 k8s-agentpool-24944505-vmss000000 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:52:45 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:52:52 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:52:52 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:52:45 +0000 UTC }] Nov 13 20:57:47.548: INFO: csi-hostpath-provisioner-0 k8s-agentpool-24944505-vmss000000 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:52:46 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:52:54 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:52:54 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:52:46 +0000 UTC }] Nov 13 20:57:47.548: INFO: csi-hostpath-resizer-0 k8s-agentpool-24944505-vmss000000 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:52:46 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:52:55 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:52:55 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:52:46 +0000 UTC }] Nov 13 20:57:47.548: INFO: csi-hostpathplugin-0 k8s-agentpool-24944505-vmss000000 Pending [] Nov 13 20:57:47.548: INFO: csi-snapshotter-0 k8s-agentpool-24944505-vmss000000 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:52:46 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:52:56 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:52:56 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2019-11-13 20:52:46 +0000 UTC }] Nov 13 20:57:47.548: INFO: Nov 13 20:57:47.664: INFO: Logging node info for node k8s-agentpool-24944505-vmss000000 Nov 13 20:57:47.722: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool-24944505-vmss000000 /api/v1/nodes/k8s-agentpool-24944505-vmss000000 5bc7cf9e-80da-4e26-a2c2-20757e870d68 27652 0 2019-11-13 20:42:13 +0000 UTC <nil> <nil> map[agentpool:agentpool beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westus2 failure-domain.beta.kubernetes.io/zone:westus2-1 kubernetes.azure.com/cluster:kubetest-74ff5bf7-0652-11ea-bade-36185e316941 kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool-24944505-vmss000000 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: storageprofile:managed storagetier:Premium_LRS] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-1137":"k8s-agentpool-24944505-vmss000000","csi-hostpath-ephemeral-9135":"k8s-agentpool-24944505-vmss000000","csi-hostpath-provisioning-716":"k8s-agentpool-24944505-vmss000000","csi-hostpath-volume-expand-4561":"k8s-agentpool-24944505-vmss000000","csi-hostpath-volume-expand-9975":"k8s-agentpool-24944505-vmss000000","csi-mock-csi-mock-volumes-2863":"csi-mock-csi-mock-volumes-2863","csi-mock-csi-mock-volumes-5360":"csi-mock-csi-mock-volumes-5360","csi-mock-csi-mock-volumes-8606":"csi-mock-csi-mock-volumes-8606"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUse_ExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-74ff5bf7-0652-11ea-bade-36185e316941/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-24944505-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{31036776448 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16797569024 0} {<nil>} 16403876Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{27933098757 0} {<nil>} 27933098757 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16011137024 0} {<nil>} 15635876Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2019-11-13 20:42:58 +0000 UTC,LastTransitionTime:2019-11-13 20:42:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2019-11-13 20:57:47 +0000 UTC,LastTransitionTime:2019-11-13 20:42:03 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2019-11-13 20:57:4