This job view page is being replaced by Spyglass soon. Check out the new job view.
ResultFAILURE
Tests 1 failed / 69 succeeded
Started2020-03-27 02:43
Elapsed21m33s
Revision
Buildera4f21ea6-6fd4-11ea-bd6a-f29ce8b0e481
links{u'resultstore': {u'url': u'https://source.cloud.google.com/results/invocations/fc56dd3f-d1e8-4feb-95ac-f8d80631e5e1/targets/test'}}
resultstorehttps://source.cloud.google.com/results/invocations/fc56dd3f-d1e8-4feb-95ac-f8d80631e5e1/targets/test
infra-commitd2e2e04d1
job-versionv1.16.9-beta.0.17+bb616cba1cc983
master_os_imagecos-73-11647-163-0
node_os_imagecos-73-11647-163-0
revisionv1.16.9-beta.0.17+bb616cba1cc983

Test Failures


listResources After 1m22s

Failed to list resources (error during ./cluster/gce/list-resources.sh: exit status 2):
Project: k8s-jkns-gce-serial-1-3
Region: 
Zone: 
Instance prefix: bootstrap-e2e
Network: bootstrap-e2e
Provider: gce


[ compute instance-templates ]



[ compute instance-groups ]



[ compute instances ]

				from junit_runner.xml

Filter through log files | View test history on testgrid


Show 69 Passed Tests

Show 4678 Skipped Tests

Error lines from build-log.txt

... skipping 15 lines ...
I0327 02:43:09.228] process 49 exited with code 0 after 0.0m
I0327 02:43:09.228] Will upload results to gs://kubernetes-jenkins/logs using pr-kubekins@kubernetes-jenkins-pull.iam.gserviceaccount.com
I0327 02:43:09.228] Root: /workspace
I0327 02:43:09.228] cd to /workspace
I0327 02:43:09.228] Configure environment...
I0327 02:43:09.229] Call:  git show -s --format=format:%ct HEAD
W0327 02:43:09.233] fatal: not a git repository (or any of the parent directories): .git
I0327 02:43:09.233] process 63 exited with code 128 after 0.0m
W0327 02:43:09.233] Unable to print commit date for HEAD
I0327 02:43:09.233] Call:  gcloud auth activate-service-account --key-file=/etc/service-account/service-account.json
W0327 02:43:09.887] Activated service account credentials for: [pr-kubekins@kubernetes-jenkins-pull.iam.gserviceaccount.com]
I0327 02:43:10.036] process 64 exited with code 0 after 0.0m
I0327 02:43:10.037] Call:  gcloud config get-value account
... skipping 202 lines ...
W0327 02:44:09.381] Project: k8s-jkns-gce-serial-1-3
W0327 02:44:09.381] Network Project: k8s-jkns-gce-serial-1-3
W0327 02:44:09.382] Zone: us-west1-b
W0327 02:44:11.532] INSTANCE_GROUPS=
W0327 02:44:11.533] NODE_NAMES=
I0327 02:44:11.633] Bringing down cluster
W0327 02:44:13.992] ERROR: (gcloud.compute.instances.list) Some requests did not succeed:
W0327 02:44:13.993]  - Invalid value for field 'zone': 'us-west4-a'. Unknown zone.
W0327 02:44:13.993]  - Invalid value for field 'zone': 'us-west4-b'. Unknown zone.
W0327 02:44:13.993]  - Invalid value for field 'zone': 'us-west4-c'. Unknown zone.
W0327 02:44:13.993] 
I0327 02:44:36.747] Deleting firewall rules remaining in network bootstrap-e2e: 
W0327 02:44:39.752] W0327 02:44:39.752344    1223 loader.go:223] Config not found: /workspace/.kube/config
... skipping 162 lines ...
W0327 02:47:26.989] Trying to find master named 'bootstrap-e2e-master'
W0327 02:47:26.989] Looking for address 'bootstrap-e2e-master-ip'
W0327 02:47:28.054] Using master: bootstrap-e2e-master (external IP: 35.233.154.176; internal IP: (not set))
I0327 02:47:28.155] Waiting up to 300 seconds for cluster initialization.
I0327 02:47:28.155] 
I0327 02:47:28.155]   This will continually check to see if the API for kubernetes is reachable.
I0327 02:47:28.155]   This may time out if there was some uncaught error during start up.
I0327 02:47:28.155] 
I0327 02:48:13.863] ............Kubernetes cluster created.
I0327 02:48:14.041] Cluster "k8s-jkns-gce-serial-1-3_bootstrap-e2e" set.
I0327 02:48:14.226] User "k8s-jkns-gce-serial-1-3_bootstrap-e2e" set.
I0327 02:48:14.403] Context "k8s-jkns-gce-serial-1-3_bootstrap-e2e" created.
I0327 02:48:14.618] Switched to context "k8s-jkns-gce-serial-1-3_bootstrap-e2e".
... skipping 135 lines ...
I0327 02:49:34.414] 
I0327 02:49:52.056] Mar 27 02:49:39.488: INFO: cluster-master-image: cos-73-11647-163-0
I0327 02:49:52.057] Mar 27 02:49:39.488: INFO: cluster-node-image: cos-73-11647-163-0
I0327 02:49:52.057] Mar 27 02:49:39.488: INFO: >>> kubeConfig: /workspace/.kube/config
I0327 02:49:52.057] Mar 27 02:49:39.492: INFO: Waiting up to 30m0s for all (but 0) nodes to be schedulable
I0327 02:49:52.058] Mar 27 02:49:39.639: INFO: Waiting up to 10m0s for all pods (need at least 8) in namespace 'kube-system' to be running and ready
I0327 02:49:52.058] Mar 27 02:49:39.804: INFO: The status of Pod fluentd-gcp-v3.2.0-6zn62 is Pending (Ready = false), waiting for it to be either Running (with Ready = true) or Failed
I0327 02:49:52.062] Mar 27 02:49:39.804: INFO: 27 / 28 pods in namespace 'kube-system' are running and ready (0 seconds elapsed)
I0327 02:49:52.063] Mar 27 02:49:39.804: INFO: expected 9 pod replicas in namespace 'kube-system', 9 are Running and Ready.
I0327 02:49:52.063] Mar 27 02:49:39.804: INFO: POD                       NODE                             PHASE    GRACE  CONDITIONS
I0327 02:49:52.064] Mar 27 02:49:39.804: INFO: fluentd-gcp-v3.2.0-6zn62  bootstrap-e2e-minion-group-r4cj  Pending  60s    [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2020-03-27 02:48:38 +0000 UTC  } {Ready False 0001-01-01 00:00:00 +0000 UTC 2020-03-27 02:49:36 +0000 UTC ContainersNotReady containers with unready status: [fluentd-gcp prometheus-to-sd-exporter]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2020-03-27 02:49:36 +0000 UTC ContainersNotReady containers with unready status: [fluentd-gcp prometheus-to-sd-exporter]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2020-03-27 02:48:38 +0000 UTC  }]
I0327 02:49:52.065] Mar 27 02:49:39.804: INFO: 
I0327 02:49:52.065] Mar 27 02:49:41.918: INFO: The status of Pod fluentd-gcp-v3.2.0-6zn62 is Pending (Ready = false), waiting for it to be either Running (with Ready = true) or Failed
I0327 02:49:52.065] Mar 27 02:49:41.918: INFO: 27 / 28 pods in namespace 'kube-system' are running and ready (2 seconds elapsed)
I0327 02:49:52.071] Mar 27 02:49:41.918: INFO: expected 9 pod replicas in namespace 'kube-system', 9 are Running and Ready.
I0327 02:49:52.072] Mar 27 02:49:41.918: INFO: POD                       NODE                             PHASE    GRACE  CONDITIONS
I0327 02:49:52.073] Mar 27 02:49:41.918: INFO: fluentd-gcp-v3.2.0-6zn62  bootstrap-e2e-minion-group-r4cj  Pending  60s    [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2020-03-27 02:48:38 +0000 UTC  } {Ready False 0001-01-01 00:00:00 +0000 UTC 2020-03-27 02:49:36 +0000 UTC ContainersNotReady containers with unready status: [fluentd-gcp prometheus-to-sd-exporter]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2020-03-27 02:49:36 +0000 UTC ContainersNotReady containers with unready status: [fluentd-gcp prometheus-to-sd-exporter]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2020-03-27 02:48:38 +0000 UTC  }]
I0327 02:49:52.073] Mar 27 02:49:41.918: INFO: 
I0327 02:49:52.073] Mar 27 02:49:43.924: INFO: The status of Pod fluentd-gcp-v3.2.0-6zn62 is Pending (Ready = false), waiting for it to be either Running (with Ready = true) or Failed
I0327 02:49:52.074] Mar 27 02:49:43.924: INFO: 27 / 28 pods in namespace 'kube-system' are running and ready (4 seconds elapsed)
I0327 02:49:52.079] Mar 27 02:49:43.924: INFO: expected 9 pod replicas in namespace 'kube-system', 9 are Running and Ready.
I0327 02:49:52.079] Mar 27 02:49:43.924: INFO: POD                       NODE                             PHASE    GRACE  CONDITIONS
I0327 02:49:52.080] Mar 27 02:49:43.924: INFO: fluentd-gcp-v3.2.0-6zn62  bootstrap-e2e-minion-group-r4cj  Pending  60s    [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2020-03-27 02:48:38 +0000 UTC  } {Ready False 0001-01-01 00:00:00 +0000 UTC 2020-03-27 02:49:36 +0000 UTC ContainersNotReady containers with unready status: [fluentd-gcp prometheus-to-sd-exporter]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2020-03-27 02:49:36 +0000 UTC ContainersNotReady containers with unready status: [fluentd-gcp prometheus-to-sd-exporter]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2020-03-27 02:48:38 +0000 UTC  }]
I0327 02:49:52.080] Mar 27 02:49:43.924: INFO: 
I0327 02:49:52.081] Mar 27 02:49:45.924: INFO: The status of Pod fluentd-gcp-v3.2.0-6zn62 is Pending (Ready = false), waiting for it to be either Running (with Ready = true) or Failed
I0327 02:49:52.081] Mar 27 02:49:45.924: INFO: 27 / 28 pods in namespace 'kube-system' are running and ready (6 seconds elapsed)
I0327 02:49:52.081] Mar 27 02:49:45.924: INFO: expected 9 pod replicas in namespace 'kube-system', 9 are Running and Ready.
I0327 02:49:52.082] Mar 27 02:49:45.924: INFO: POD                       NODE                             PHASE    GRACE  CONDITIONS
I0327 02:49:52.087] Mar 27 02:49:45.924: INFO: fluentd-gcp-v3.2.0-6zn62  bootstrap-e2e-minion-group-r4cj  Pending  60s    [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2020-03-27 02:48:38 +0000 UTC  } {Ready False 0001-01-01 00:00:00 +0000 UTC 2020-03-27 02:49:36 +0000 UTC ContainersNotReady containers with unready status: [fluentd-gcp prometheus-to-sd-exporter]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2020-03-27 02:49:36 +0000 UTC ContainersNotReady containers with unready status: [fluentd-gcp prometheus-to-sd-exporter]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2020-03-27 02:48:38 +0000 UTC  }]
I0327 02:49:52.087] Mar 27 02:49:45.924: INFO: 
I0327 02:49:52.088] Mar 27 02:49:47.918: INFO: The status of Pod fluentd-gcp-v3.2.0-6zn62 is Pending (Ready = false), waiting for it to be either Running (with Ready = true) or Failed
I0327 02:49:52.088] Mar 27 02:49:47.918: INFO: 27 / 28 pods in namespace 'kube-system' are running and ready (8 seconds elapsed)
I0327 02:49:52.089] Mar 27 02:49:47.918: INFO: expected 9 pod replicas in namespace 'kube-system', 9 are Running and Ready.
I0327 02:49:52.089] Mar 27 02:49:47.918: INFO: POD                       NODE                             PHASE    GRACE  CONDITIONS
I0327 02:49:52.090] Mar 27 02:49:47.918: INFO: fluentd-gcp-v3.2.0-6zn62  bootstrap-e2e-minion-group-r4cj  Pending  60s    [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2020-03-27 02:48:38 +0000 UTC  } {Ready False 0001-01-01 00:00:00 +0000 UTC 2020-03-27 02:49:36 +0000 UTC ContainersNotReady containers with unready status: [fluentd-gcp prometheus-to-sd-exporter]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2020-03-27 02:49:36 +0000 UTC ContainersNotReady containers with unready status: [fluentd-gcp prometheus-to-sd-exporter]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2020-03-27 02:48:38 +0000 UTC  }]
I0327 02:49:52.091] Mar 27 02:49:47.918: INFO: 
I0327 02:49:52.091] Mar 27 02:49:49.918: INFO: The status of Pod fluentd-gcp-v3.2.0-hdvsg is Pending (Ready = false), waiting for it to be either Running (with Ready = true) or Failed
I0327 02:49:52.096] Mar 27 02:49:49.918: INFO: 27 / 28 pods in namespace 'kube-system' are running and ready (10 seconds elapsed)
I0327 02:49:52.096] Mar 27 02:49:49.918: INFO: expected 9 pod replicas in namespace 'kube-system', 9 are Running and Ready.
I0327 02:49:52.096] Mar 27 02:49:49.918: INFO: POD                       NODE                             PHASE    GRACE  CONDITIONS
I0327 02:49:52.099] Mar 27 02:49:49.918: INFO: fluentd-gcp-v3.2.0-hdvsg  bootstrap-e2e-minion-group-r4cj  Pending         [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2020-03-27 02:49:48 +0000 UTC  } {Ready False 0001-01-01 00:00:00 +0000 UTC 2020-03-27 02:49:48 +0000 UTC ContainersNotReady containers with unready status: [fluentd-gcp prometheus-to-sd-exporter]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2020-03-27 02:49:48 +0000 UTC ContainersNotReady containers with unready status: [fluentd-gcp prometheus-to-sd-exporter]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2020-03-27 02:49:48 +0000 UTC  }]
I0327 02:49:52.099] Mar 27 02:49:49.918: INFO: 
I0327 02:49:52.102] Mar 27 02:49:51.924: INFO: 28 / 28 pods in namespace 'kube-system' are running and ready (12 seconds elapsed)
... skipping 894 lines ...
I0327 02:50:34.741]   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl/kubectl.go:225
I0327 02:50:34.742] [It] should check if cluster-info dump succeeds
I0327 02:50:34.742]   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl/kubectl.go:994
I0327 02:50:34.742] STEP: running cluster-info dump
I0327 02:50:34.742] Mar 27 02:50:23.139: INFO: Running '../../../../kubernetes_skew/cluster/kubectl.sh --server=https://35.233.154.176 --kubeconfig=/workspace/.kube/config cluster-info dump'
I0327 02:50:34.743] Mar 27 02:50:27.138: INFO: stderr: ""
I0327 02:50:35.698] Mar 27 02:50:27.139: INFO: stdout: "{\n    \"kind\": \"NodeList\",\n    \"apiVersion\": \"v1\",\n    \"metadata\": {\n        \"selfLink\": \"/api/v1/nodes\",\n        \"resourceVersion\": \"1577\"\n    },\n    \"items\": [\n        {\n            \"metadata\": {\n                \"name\": \"bootstrap-e2e-master\",\n                \"selfLink\": \"/api/v1/nodes/bootstrap-e2e-master\",\n                \"uid\": \"fad7e742-343f-4852-a7e5-132b4ca8d259\",\n                \"resourceVersion\": \"1552\",\n                \"creationTimestamp\": \"2020-03-27T02:48:41Z\",\n                \"labels\": {\n                    \"beta.kubernetes.io/arch\": \"amd64\",\n                    \"beta.kubernetes.io/instance-type\": \"n1-standard-1\",\n                    \"beta.kubernetes.io/os\": \"linux\",\n                    \"cloud.google.com/metadata-proxy-ready\": \"true\",\n                    \"failure-domain.beta.kubernetes.io/region\": \"us-west1\",\n                    \"failure-domain.beta.kubernetes.io/zone\": \"us-west1-b\",\n                    \"kubernetes.io/arch\": \"amd64\",\n                    \"kubernetes.io/hostname\": \"bootstrap-e2e-master\",\n                    \"kubernetes.io/os\": \"linux\"\n                },\n                \"annotations\": {\n                    \"node.alpha.kubernetes.io/ttl\": \"0\",\n                    \"volumes.kubernetes.io/controller-managed-attach-detach\": \"true\"\n                }\n            },\n            \"spec\": {\n                \"podCIDR\": \"10.64.3.0/24\",\n                \"podCIDRs\": [\n                    \"10.64.3.0/24\"\n                ],\n                \"providerID\": \"gce://k8s-jkns-gce-serial-1-3/us-west1-b/bootstrap-e2e-master\",\n                \"unschedulable\": true,\n                \"taints\": [\n                    {\n                        \"key\": \"node-role.kubernetes.io/master\",\n                        \"effect\": \"NoSchedule\"\n                    },\n                    {\n                        \"key\": \"node.kubernetes.io/unschedulable\",\n                        \"effect\": \"NoSchedule\"\n                    }\n                ]\n            },\n            \"status\": {\n                \"capacity\": {\n                    \"attachable-volumes-gce-pd\": \"127\",\n                    \"cpu\": \"1\",\n                    \"ephemeral-storage\": \"16293736Ki\",\n                    \"hugepages-2Mi\": \"0\",\n                    \"memory\": \"3787524Ki\",\n                    \"pods\": \"110\"\n                },\n                \"allocatable\": {\n                    \"attachable-volumes-gce-pd\": \"127\",\n                    \"cpu\": \"1\",\n                    \"ephemeral-storage\": \"15016307073\",\n                    \"hugepages-2Mi\": \"0\",\n                    \"memory\": \"3531524Ki\",\n                    \"pods\": \"110\"\n                },\n                \"conditions\": [\n                    {\n                        \"type\": \"NetworkUnavailable\",\n                        \"status\": \"False\",\n                        \"lastHeartbeatTime\": \"2020-03-27T02:48:55Z\",\n                        \"lastTransitionTime\": \"2020-03-27T02:48:55Z\",\n                        \"reason\": \"RouteCreated\",\n                        \"message\": \"RouteController created a route\"\n                    },\n                    {\n                        \"type\": \"MemoryPressure\",\n                        \"status\": \"False\",\n                        \"lastHeartbeatTime\": \"2020-03-27T02:50:22Z\",\n                        \"lastTransitionTime\": \"2020-03-27T02:48:41Z\",\n                        \"reason\": \"KubeletHasSufficientMemory\",\n                        \"message\": \"kubelet has sufficient memory available\"\n                    },\n                    {\n                        \"type\": \"DiskPressure\",\n                        \"status\": \"False\",\n                        \"lastHeartbeatTime\": \"2020-03-27T02:50:22Z\",\n                        \"lastTransitionTime\": \"2020-03-27T02:48:41Z\",\n                        \"reason\": \"KubeletHasNoDiskPressure\",\n                        \"message\": \"kubelet has no disk pressure\"\n                    },\n                    {\n                        \"type\": \"PIDPressure\",\n                        \"status\": \"False\",\n                        \"lastHeartbeatTime\": \"2020-03-27T02:50:22Z\",\n                        \"lastTransitionTime\": \"2020-03-27T02:48:41Z\",\n                        \"reason\": \"KubeletHasSufficientPID\",\n                        \"message\": \"kubelet has sufficient PID available\"\n                    },\n                    {\n                        \"type\": \"Ready\",\n                        \"status\": \"True\",\n                        \"lastHeartbeatTime\": \"2020-03-27T02:50:22Z\",\n                        \"lastTransitionTime\": \"2020-03-27T02:48:41Z\",\n                        \"reason\": \"KubeletReady\",\n                        \"message\": \"kubelet is posting ready status. AppArmor enabled\"\n                    }\n                ],\n                \"addresses\": [\n                    {\n                        \"type\": \"InternalIP\",\n                        \"address\": \"10.138.0.2\"\n                    },\n                    {\n                        \"type\": \"ExternalIP\",\n                        \"address\": \"35.233.154.176\"\n                    },\n                    {\n                        \"type\": \"InternalDNS\",\n                        \"address\": \"bootstrap-e2e-master.c.k8s-jkns-gce-serial-1-3.internal\"\n                    },\n                    {\n                        \"type\": \"Hostname\",\n                        \"address\": \"bootstrap-e2e-master.c.k8s-jkns-gce-serial-1-3.internal\"\n                    }\n                ],\n                \"daemonEndpoints\": {\n                    \"kubeletEndpoint\": {\n                        \"Port\": 10250\n                    }\n                },\n                \"nodeInfo\": {\n                    \"machineID\": \"6fe3bf907ccaa9deadea480c067f2604\",\n                    \"systemUUID\": \"6FE3BF90-7CCA-A9DE-ADEA-480C067F2604\",\n                    \"bootID\": \"53ede7b4-77d3-4e6e-ac81-98d2c83f6357\",\n                    \"kernelVersion\": \"4.14.94+\",\n                    \"osImage\": \"Container-Optimized OS from Google\",\n                    \"containerRuntimeVersion\": \"docker://18.9.3\",\n                    \"kubeletVersion\": \"v1.16.9-beta.0.17+bb616cba1cc983\",\n                    \"kubeProxyVersion\": \"v1.16.9-beta.0.17+bb616cba1cc983\",\n                    \"operatingSystem\": \"linux\",\n                    \"architecture\": \"amd64\"\n                },\n                \"images\": [\n                    {\n                        \"names\": [\n                            \"gcr.io/stackdriver-agents/stackdriver-logging-agent@sha256:6c8574a40816676cd908cfa89d16463002b56ca05fa76d0c912e116bc0ab867e\",\n                            \"gcr.io/stackdriver-agents/stackdriver-logging-agent:1.6.8\"\n                        ],\n                        \"sizeBytes\": 264721247\n                    },\n                    {\n                        \"names\": [\n                            \"k8s.gcr.io/etcd@sha256:12c2c5e5731c3bcd56e6f1c05c0f9198b6f06793fa7fca2fb43aab9622dc4afa\",\n                            \"k8s.gcr.io/etcd:3.3.15-0\"\n                        ],\n                        \"sizeBytes\": 246640776\n                    },\n                    {\n                        \"names\": [\n                            \"k8s.gcr.io/kube-apiserver-amd64:v1.16.9-beta.0.17_bb616cba1cc983\"\n                        ],\n                        \"sizeBytes\": 200393484\n                    },\n                    {\n                        \"names\": [\n                            \"k8s.gcr.io/kube-controller-manager-amd64:v1.16.9-beta.0.17_bb616cba1cc983\"\n                        ],\n                        \"sizeBytes\": 191496669\n                    },\n                    {\n                        \"names\": [\n                            \"k8s.gcr.io/kube-scheduler-amd64:v1.16.9-beta.0.17_bb616cba1cc983\"\n                        ],\n                        \"sizeBytes\": 99218726\n                    },\n                    {\n                        \"names\": [\n                            \"k8s.gcr.io/kube-addon-manager@sha256:3e315022a842d782a28e729720f21091dde21f1efea28868d65ec595ad871616\",\n                            \"k8s.gcr.io/kube-addon-manager:v9.0.2\"\n                        ],\n                        \"sizeBytes\": 83076028\n                    },\n                    {\n                        \"names\": [\n                            \"k8s.gcr.io/etcd-empty-dir-cleanup@sha256:c60552d15b383b88a75d552816041cfe963edc3e4d9e8e378d7c0aae20cfb3f7\",\n                            \"k8s.gcr.io/etcd-empty-dir-cleanup:3.3.15.0\"\n                        ],\n                        \"sizeBytes\": 78265220\n                    },\n                    {\n                        \"names\": [\n                            \"k8s.gcr.io/ingress-gce-glbc-amd64@sha256:14f14351a03038b238232e60850a9cfa0dffbed0590321ef84216a432accc1ca\",\n                            \"k8s.gcr.io/ingress-gce-glbc-amd64:v1.2.3\"\n                        ],\n                        \"sizeBytes\": 71797285\n                    },\n                    {\n                        \"names\": [\n                            \"k8s.gcr.io/prometheus-to-sd@sha256:14666989f40bb7c896c3e775a93c6873e2b791d65bc65579f58a078b7f9a764e\",\n                            \"k8s.gcr.io/prometheus-to-sd:v0.5.0\"\n                        ],\n                        \"sizeBytes\": 41861013\n                    },\n                    {\n                        \"names\": [\n                            \"k8s.gcr.io/metadata-proxy@sha256:e914645f22e946bce5165737e1b244e0a296ad1f0f81a9531adc57af2780978a\",\n                            \"k8s.gcr.io/metadata-proxy:v0.1.12\"\n                        ],\n                        \"sizeBytes\": 11337839\n                    },\n                    {\n                        \"names\": [\n                            \"k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea\",\n                            \"k8s.gcr.io/pause:3.1\"\n                        ],\n                        \"sizeBytes\": 742472\n                    }\n                ],\n                \"config\": {}\n            }\n        },\n        {\n            \"metadata\": {\n                \"name\": \"bootstrap-e2e-minion-group-0m0b\",\n                \"selfLink\": \"/api/v1/nodes/bootstrap-e2e-minion-group-0m0b\",\n                \"uid\": \"4681e39c-1b90-4c0b-9892-7cc60115e162\",\n                \"resourceVersion\": \"1325\",\n                \"creationTimestamp\": \"2020-03-27T02:48:38Z\",\n                \"labels\": {\n                    \"beta.kubernetes.io/arch\": \"amd64\",\n                    \"beta.kubernetes.io/instance-type\": \"n1-standard-2\",\n                    \"beta.kubernetes.io/os\": \"linux\",\n                    \"cloud.google.com/metadata-proxy-ready\": \"true\",\n                    \"failure-domain.beta.kubernetes.io/region\": \"us-west1\",\n                    \"failure-domain.beta.kubernetes.io/zone\": \"us-west1-b\",\n                    \"kubernetes.io/arch\": \"amd64\",\n                    \"kubernetes.io/hostname\": \"bootstrap-e2e-minion-group-0m0b\",\n                    \"kubernetes.io/os\": \"linux\"\n                },\n                \"annotations\": {\n                    \"node.alpha.kubernetes.io/ttl\": \"0\",\n                    \"volumes.kubernetes.io/controller-managed-attach-detach\": \"true\"\n                }\n            },\n            \"spec\": {\n                \"podCIDR\": \"10.64.2.0/24\",\n                \"podCIDRs\": [\n                    \"10.64.2.0/24\"\n                ],\n                \"providerID\": \"gce://k8s-jkns-gce-serial-1-3/us-west1-b/bootstrap-e2e-minion-group-0m0b\"\n            },\n            \"status\": {\n                \"capacity\": {\n                    \"attachable-volumes-gce-pd\": \"127\",\n                    \"cpu\": \"2\",\n                    \"ephemeral-storage\": \"98868448Ki\",\n                    \"hugepages-2Mi\": \"0\",\n                    \"memory\": \"7658068Ki\",\n                    \"pods\": \"110\"\n                },\n                \"allocatable\": {\n                    \"attachable-volumes-gce-pd\": \"127\",\n                    \"cpu\": \"2\",\n                    \"ephemeral-storage\": \"91117161526\",\n                    \"hugepages-2Mi\": \"0\",\n                    \"memory\": \"7402068Ki\",\n                    \"pods\": \"110\"\n                },\n                \"conditions\": [\n                    {\n                        \"type\": \"ReadonlyFilesystem\",\n                        \"status\": \"False\",\n                        \"lastHeartbeatTime\": \"2020-03-27T02:49:44Z\",\n                        \"lastTransitionTime\": \"2020-03-27T02:48:43Z\",\n                        \"reason\": \"FilesystemIsNotReadOnly\",\n                        \"message\": \"Filesystem is not read-only\"\n                    },\n                    {\n                        \"type\": \"CorruptDockerOverlay2\",\n                        \"status\": \"False\",\n                        \"lastHeartbeatTime\": \"2020-03-27T02:49:44Z\",\n                        \"lastTransitionTime\": \"2020-03-27T02:48:43Z\",\n                        \"reason\": \"NoCorruptDockerOverlay2\",\n                        \"message\": \"docker overlay2 is functioning properly\"\n                    },\n                    {\n                        \"type\": \"FrequentUnregisterNetDevice\",\n                        \"status\": \"False\",\n                        \"lastHeartbeatTime\": \"2020-03-27T02:49:44Z\",\n                        \"lastTransitionTime\": \"2020-03-27T02:48:43Z\",\n                        \"reason\": \"NoFrequentUnregisterNetDevice\",\n                        \"message\": \"node is functioning properly\"\n                    },\n                    {\n                        \"type\": \"FrequentKubeletRestart\",\n                        \"status\": \"False\",\n                        \"lastHeartbeatTime\": \"2020-03-27T02:49:44Z\",\n                        \"lastTransitionTime\": \"2020-03-27T02:48:43Z\",\n                        \"reason\": \"NoFrequentKubeletRestart\",\n                        \"message\": \"kubelet is functioning properly\"\n                    },\n                    {\n                        \"type\": \"FrequentDockerRestart\",\n                        \"status\": \"False\",\n                        \"lastHeartbeatTime\": \"2020-03-27T02:49:44Z\",\n                        \"lastTransitionTime\": \"2020-03-27T02:48:43Z\",\n                        \"reason\": \"NoFrequentDockerRestart\",\n                        \"message\": \"docker is functioning properly\"\n                    },\n                    {\n                        \"type\": \"FrequentContainerdRestart\",\n                        \"status\": \"False\",\n                        \"lastHeartbeatTime\": \"2020-03-27T02:49:44Z\",\n                        \"lastTransitionTime\": \"2020-03-27T02:48:43Z\",\n                        \"reason\": \"NoFrequentContainerdRestart\",\n                        \"message\": \"containerd is functioning properly\"\n                    },\n                    {\n                        \"type\": \"KernelDeadlock\",\n                        \"status\": \"False\",\n                        \"lastHeartbeatTime\": \"2020-03-27T02:49:44Z\",\n                        \"lastTransitionTime\": \"2020-03-27T02:48:43Z\",\n                        \"reason\": \"KernelHasNoDeadlock\",\n                        \"message\": \"kernel has no deadlock\"\n                    },\n                    {\n                        \"type\": \"NetworkUnavailable\",\n                        \"status\": \"False\",\n                        \"lastHeartbeatTime\": \"2020-03-27T02:48:46Z\",\n                        \"lastTransitionTime\": \"2020-03-27T02:48:46Z\",\n                        \"reason\": \"RouteCreated\",\n                        \"message\": \"RouteController created a route\"\n                    },\n                    {\n                        \"type\": \"MemoryPressure\",\n                        \"status\": \"False\",\n                        \"lastHeartbeatTime\": \"2020-03-27T02:50:09Z\",\n                        \"lastTransitionTime\": \"2020-03-27T02:48:38Z\",\n                        \"reason\": \"KubeletHasSufficientMemory\",\n                        \"message\": \"kubelet has sufficient memory available\"\n                    },\n                    {\n                        \"type\": \"DiskPressure\",\n                        \"status\": \"False\",\n                        \"lastHeartbeatTime\": \"2020-03-27T02:50:09Z\",\n                        \"lastTransitionTime\": \"2020-03-27T02:48:38Z\",\n                        \"reason\": \"KubeletHasNoDiskPressure\",\n                        \"message\": \"kubelet has no disk pressure\"\n                    },\n                    {\n                        \"type\": \"PIDPressure\",\n                        \"status\": \"False\",\n                        \"lastHeartbeatTime\": \"2020-03-27T02:50:09Z\",\n                        \"lastTransitionTime\": \"2020-03-27T02:48:38Z\",\n                        \"reason\": \"KubeletHasSufficientPID\",\n                        \"message\": \"kubelet has sufficient PID available\"\n                    },\n                    {\n                        \"type\": \"Ready\",\n                        \"status\": \"True\",\n                        \"lastHeartbeatTime\": \"2020-03-27T02:50:09Z\",\n                        \"lastTransitionTime\": \"2020-03-27T02:48:39Z\",\n                        \"reason\": \"KubeletReady\",\n                        \"message\": \"kubelet is posting ready status. AppArmor enabled\"\n                    }\n                ],\n                \"addresses\": [\n                    {\n                        \"type\": \"InternalIP\",\n                        \"address\": \"10.138.0.3\"\n                    },\n                    {\n                        \"type\": \"ExternalIP\",\n                        \"address\": \"34.83.164.199\"\n                    },\n                    {\n                        \"type\": \"InternalDNS\",\n                        \"address\": \"bootstrap-e2e-minion-group-0m0b.c.k8s-jkns-gce-serial-1-3.internal\"\n                    },\n                    {\n                        \"type\": \"Hostname\",\n                        \"address\": \"bootstrap-e2e-minion-group-0m0b.c.k8s-jkns-gce-serial-1-3.internal\"\n                    }\n                ],\n                \"daemonEndpoints\": {\n                    \"kubeletEndpoint\": {\n                        \"Port\": 10250\n                    }\n                },\n                \"nodeInfo\": {\n                    \"machineID\": \"6d3e07e8069255b836a98aa3d69d0b4d\",\n                    \"systemUUID\": \"6D3E07E8-0692-55B8-36A9-8AA3D69D0B4D\",\n                    \"bootID\": \"ead1d895-7d58-4831-bebb-e41f5053d0a6\",\n                    \"kernelVersion\": \"4.14.94+\",\n                    \"osImage\": \"Container-Optimized OS from Google\",\n                    \"containerRuntimeVersion\": \"docker://18.9.3\",\n                    \"kubeletVersion\": \"v1.16.9-beta.0.17+bb616cba1cc983\",\n                    \"kubeProxyVersion\": \"v1.16.9-beta.0.17+bb616cba1cc983\",\n                    \"operatingSystem\": \"linux\",\n                    \"architecture\": \"amd64\"\n                },\n                \"images\": [\n                    {\n                        \"names\": [\n                            \"gcr.io/stackdriver-agents/stackdriver-logging-agent@sha256:6c8574a40816676cd908cfa89d16463002b56ca05fa76d0c912e116bc0ab867e\",\n                            \"gcr.io/stackdriver-agents/stackdriver-logging-agent:1.6.8\"\n                        ],\n                        \"sizeBytes\": 264721247\n                    },\n                    {\n                        \"names\": [\n                            \"k8s.gcr.io/kube-proxy-amd64:v1.16.9-beta.0.17_bb616cba1cc983\"\n                        ],\n                        \"sizeBytes\": 96741051\n                    },\n                    {\n                        \"names\": [\n                            \"gcr.io/kubernetes-e2e-test-images/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727\",\n                            \"gcr.io/kubernetes-e2e-test-images/agnhost:2.6\"\n                        ],\n                        \"sizeBytes\": 57345321\n                    },\n                    {\n                        \"names\": [\n                            \"k8s.gcr.io/prometheus-to-sd@sha256:14666989f40bb7c896c3e775a93c6873e2b791d65bc65579f58a078b7f9a764e\",\n                            \"k8s.gcr.io/prometheus-to-sd:v0.5.0\"\n                        ],\n                        \"sizeBytes\": 41861013\n                    },\n                    {\n                        \"names\": [\n                            \"k8s.gcr.io/cluster-proportional-autoscaler-amd64@sha256:a2db01cfd2ae1a16f0feef274160c659c1ac5aa433e1c514de20e334cb66c674\",\n                            \"k8s.gcr.io/cluster-proportional-autoscaler-amd64:1.7.1\"\n                        ],\n                        \"sizeBytes\": 40067731\n                    },\n                    {\n                        \"names\": [\n                            \"k8s.gcr.io/metrics-server-amd64@sha256:c9c4e95068b51d6b33a9dccc61875df07dc650abbf4ac1a19d58b4628f89288b\",\n                            \"k8s.gcr.io/metrics-server-amd64:v0.3.6\"\n                        ],\n                        \"sizeBytes\": 39944451\n                    },\n                    {\n                        \"names\": [\n                            \"k8s.gcr.io/addon-resizer@sha256:30b3b12e471c534949e12d2da958fdf33848d153f2a0a88565bdef7ca999b5ad\",\n                            \"k8s.gcr.io/addon-resizer:1.8.7\"\n                        ],\n                        \"sizeBytes\": 37930718\n                    },\n                    {\n                        \"names\": [\n                            \"k8s.gcr.io/metadata-proxy@sha256:e914645f22e946bce5165737e1b244e0a296ad1f0f81a9531adc57af2780978a\",\n                            \"k8s.gcr.io/metadata-proxy:v0.1.12\"\n                        ],\n                        \"sizeBytes\": 11337839\n                    },\n                    {\n                        \"names\": [\n                            \"gcr.io/kubernetes-e2e-test-images/nautilus@sha256:33a732d4c42a266912a5091598a0f07653c9134db4b8d571690d8afd509e0bfc\",\n                            \"gcr.io/kubernetes-e2e-test-images/nautilus:1.0\"\n                        ],\n                        \"sizeBytes\": 4753501\n                    },\n                    {\n                        \"names\": [\n                            \"k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea\",\n                            \"k8s.gcr.io/pause:3.1\"\n                        ],\n                        \"sizeBytes\": 742472\n                    }\n                ],\n                \"config\": {}\n            }\n        },\n        {\n            \"metadata\": {\n                \"name\": \"bootstrap-e2e-minion-group-r4cj\",\n                \"selfLink\": \"/api/v1/nodes/bootstrap-e2e-minion-group-r4cj\",\n                \"uid\": \"b1bf888a-207a-458c-a45e-0310d8255bf8\",\n                \"resourceVersion\": \"1323\",\n                \"creationTimestamp\": \"2020-03-27T02:48:38Z\",\n                \"labels\": {\n                    \"beta.kubernetes.io/arch\": \"amd64\",\n                    \"beta.kubernetes.io/instance-type\": \"n1-standard-2\",\n                    \"beta.kubernetes.io/os\": \"linux\",\n                    \"cloud.google.com/metadata-proxy-ready\": \"true\",\n                    \"failure-domain.beta.kubernetes.io/region\": \"us-west1\",\n                    \"failure-domain.beta.kubernetes.io/zone\": \"us-west1-b\",\n                    \"kubernetes.io/arch\": \"amd64\",\n                    \"kubernetes.io/hostname\": \"bootstrap-e2e-minion-group-r4cj\",\n                    \"kubernetes.io/os\": \"linux\"\n                },\n                \"annotations\": {\n                    \"node.alpha.kubernetes.io/ttl\": \"0\",\n                    \"volumes.kubernetes.io/controller-managed-attach-detach\": \"true\"\n                }\n            },\n            \"spec\": {\n                \"podCIDR\": \"10.64.1.0/24\",\n                \"podCIDRs\": [\n                    \"10.64.1.0/24\"\n                ],\n                \"providerID\": \"gce://k8s-jkns-gce-serial-1-3/us-west1-b/bootstrap-e2e-minion-group-r4cj\"\n            },\n            \"status\": {\n                \"capacity\": {\n                    \"attachable-volumes-gce-pd\": \"127\",\n                    \"cpu\": \"2\",\n                    \"ephemeral-storage\": \"98868448Ki\",\n                    \"hugepages-2Mi\": \"0\",\n                    \"memory\": \"7658060Ki\",\n                    \"pods\": \"110\"\n                },\n                \"allocatable\": {\n                    \"attachable-volumes-gce-pd\": \"127\",\n                    \"cpu\": \"2\",\n                    \"ephemeral-storage\": \"91117161526\",\n                    \"hugepages-2Mi\": \"0\",\n                    \"memory\": \"7402060Ki\",\n                    \"pods\": \"110\"\n                },\n                \"conditions\": [\n                    {\n                        \"type\": \"FrequentUnregisterNetDevice\",\n                        \"status\": \"False\",\n                        \"lastHeartbeatTime\": \"2020-03-27T02:49:41Z\",\n                        \"lastTransitionTime\": \"2020-03-27T02:48:39Z\",\n                        \"reason\": \"NoFrequentUnregisterNetDevice\",\n                        \"message\": \"node is functioning properly\"\n                    },\n                    {\n                        \"type\": \"FrequentKubeletRestart\",\n                        \"status\": \"False\",\n                        \"lastHeartbeatTime\": \"2020-03-27T02:49:41Z\",\n                        \"lastTransitionTime\": \"2020-03-27T02:48:39Z\",\n                        \"reason\": \"NoFrequentKubeletRestart\",\n                        \"message\": \"kubelet is functioning properly\"\n                    },\n                    {\n                        \"type\": \"FrequentDockerRestart\",\n                        \"status\": \"False\",\n                        \"lastHeartbeatTime\": \"2020-03-27T02:49:41Z\",\n                        \"lastTransitionTime\": \"2020-03-27T02:48:39Z\",\n                        \"reason\": \"NoFrequentDockerRestart\",\n                        \"message\": \"docker is functioning properly\"\n                    },\n                    {\n                        \"type\": \"FrequentContainerdRestart\",\n                        \"status\": \"False\",\n                        \"lastHeartbeatTime\": \"2020-03-27T02:49:41Z\",\n                        \"lastTransitionTime\": \"2020-03-27T02:48:39Z\",\n                        \"reason\": \"NoFrequentContainerdRestart\",\n                        \"message\": \"containerd is functioning properly\"\n                    },\n                    {\n                        \"type\": \"KernelDeadlock\",\n                        \"status\": \"False\",\n                        \"lastHeartbeatTime\": \"2020-03-27T02:49:41Z\",\n                        \"lastTransitionTime\": \"2020-03-27T02:48:39Z\",\n                        \"reason\": \"KernelHasNoDeadlock\",\n                        \"message\": \"kernel has no deadlock\"\n                    },\n                    {\n                        \"type\": \"ReadonlyFilesystem\",\n                        \"status\": \"False\",\n                        \"lastHeartbeatTime\": \"2020-03-27T02:49:41Z\",\n                        \"lastTransitionTime\": \"2020-03-27T02:48:39Z\",\n                        \"reason\": \"FilesystemIsNotReadOnly\",\n                        \"message\": \"Filesystem is not read-only\"\n                    },\n                    {\n                        \"type\": \"CorruptDockerOverlay2\",\n                        \"status\": \"False\",\n                        \"lastHeartbeatTime\": \"2020-03-27T02:49:41Z\",\n                        \"lastTransitionTime\": \"2020-03-27T02:48:39Z\",\n                        \"reason\": \"NoCorruptDockerOverlay2\",\n                        \"message\": \"docker overlay2 is functioning properly\"\n                    },\n                    {\n                        \"type\": \"NetworkUnavailable\",\n                        \"status\": \"False\",\n                        \"lastHeartbeatTime\": \"2020-03-27T02:48:46Z\",\n                        \"lastTransitionTime\": \"2020-03-27T02:48:46Z\",\n                        \"reason\": \"RouteCreated\",\n                        \"message\": \"RouteController created a route\"\n                    },\n                    {\n                        \"type\": \"MemoryPressure\",\n                        \"status\": \"False\",\n                        \"lastHeartbeatTime\": \"2020-03-27T02:50:09Z\",\n                        \"lastTransitionTime\": \"2020-03-27T02:48:38Z\"