This job view page is being replaced by Spyglass soon. Check out the new job view.
PRvkhromov: kubeadm: mount K8s certs dir into scheduler pods
ResultFAILURE
Tests 1 failed / 1 succeeded
Started2019-07-12 00:42
Elapsed9m35s
Revision
Buildergke-prow-ssd-pool-1a225945-08s9
Refs master:4b3b536c
80064:115e1095
podcee044ab-a43d-11e9-b9c8-9690084ce7d3
infra-commita77f55c55
job-versionv1.16.0-alpha.0.2165+116bb6eff1b00c
podcee044ab-a43d-11e9-b9c8-9690084ce7d3
repok8s.io/kubernetes
repo-commit116bb6eff1b00cfb237b4d1fbc7a5ed10c9b2175
repos{u'k8s.io/kubernetes': u'master:4b3b536c510e930052f5c5b0c342f17fc8e605a3,80064:115e10956f950eced67713dd76637193f1a77688', u'k8s.io/perf-tests': u'master', u'k8s.io/release': u'master'}
revisionv1.16.0-alpha.0.2165+116bb6eff1b00c

Test Failures


Build 7m43s

error during make -C /go/src/k8s.io/kubernetes bazel-release: exit status 2
				from junit_runner.xml

Filter through log files | View test history on testgrid


Show 1 Passed Tests

Error lines from build-log.txt

... skipping 243 lines ...
W0712 00:47:35.718] [1,867 / 4,864] GoCompile vendor/github.com/Azure/azure-sdk-for-go/services/compute/mgmt/2019-03-01/compute/linux_amd64_stripped/go_default_library%/k8s.io/kubernetes/vendor/github.com/Azure/azure-sdk-for-go/services/compute/mgmt/2019-03-01/compute.a; 7s processwrapper-sandbox ... (8 actions, 7 running)
W0712 00:48:03.963] [2,360 / 4,864] GoCompile staging/src/k8s.io/api/core/v1/linux_amd64_stripped/go_default_library%/k8s.io/kubernetes/vendor/k8s.io/api/core/v1.a; 9s processwrapper-sandbox ... (8 actions, 7 running)
W0712 00:48:36.480] [3,008 / 4,864] GoCompile pkg/kubectl/util/linux_amd64_pure_stripped/go_default_library%/k8s.io/kubernetes/pkg/kubectl/util.a; 0s processwrapper-sandbox ... (8 actions, 7 running)
W0712 00:49:14.114] [3,597 / 4,864] GoCompile staging/src/k8s.io/legacy-cloud-providers/azure/linux_amd64_pure_stripped/go_default_library%/k8s.io/kubernetes/vendor/k8s.io/legacy-cloud-providers/azure.a; 1s processwrapper-sandbox ... (7 actions, 6 running)
W0712 00:49:57.373] [4,037 / 4,878] GoLink cmd/kube-scheduler/linux_amd64_pure_stripped/kube-scheduler; 15s processwrapper-sandbox ... (8 actions, 7 running)
W0712 00:50:46.971] [4,403 / 4,878] GoCompile pkg/kubectl/cmd/convert/linux_amd64_stripped/go_default_library%/k8s.io/kubernetes/pkg/kubectl/cmd/convert.a; 0s processwrapper-sandbox ... (8 actions, 7 running)
W0712 00:51:34.821] ERROR: /go/src/k8s.io/kubernetes/cmd/kubeadm/app/phases/controlplane/BUILD:28:1: GoCompile cmd/kubeadm/app/phases/controlplane/linux_amd64_stripped/go_default_library%/k8s.io/kubernetes/cmd/kubeadm/app/phases/controlplane.a failed (Exit 1) compile failed: error executing command 
W0712 00:51:34.823]   (cd /bazel-scratch/.cache/bazel/_bazel_prow/48d5366022b4e3197674c8d6e2bee219/sandbox/processwrapper-sandbox/4769/execroot/io_k8s_kubernetes && \
W0712 00:51:34.823]   exec env - \
W0712 00:51:34.824]     CGO_ENABLED=1 \
W0712 00:51:34.824]     GOARCH=amd64 \
W0712 00:51:34.847]     GOOS=linux \
W0712 00:51:34.847]     GOROOT=external/go_sdk \
W0712 00:51:34.847]     GOROOT_FINAL=GOROOT \
W0712 00:51:34.848]     PATH=/usr/bin:/bin \
W0712 00:51:34.860]   bazel-out/host/bin/external/io_bazel_rules_go/go/tools/builders/linux_amd64_stripped/compile -sdk external/go_sdk -installsuffix linux_amd64 -src cmd/kubeadm/app/phases/controlplane/manifests.go -src cmd/kubeadm/app/phases/controlplane/volumes.go -arc 'k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm=k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm=bazel-out/k8-fastbuild/bin/cmd/kubeadm/app/apis/kubeadm/linux_amd64_stripped/go_default_library%/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm.a=' -arc 'k8s.io/kubernetes/cmd/kubeadm/app/constants=k8s.io/kubernetes/cmd/kubeadm/app/constants=bazel-out/k8-fastbuild/bin/cmd/kubeadm/app/constants/linux_amd64_stripped/go_default_library%/k8s.io/kubernetes/cmd/kubeadm/app/constants.a=' -arc 'k8s.io/kubernetes/cmd/kubeadm/app/images=k8s.io/kubernetes/cmd/kubeadm/app/images=bazel-out/k8-fastbuild/bin/cmd/kubeadm/app/images/linux_amd64_stripped/go_default_library%/k8s.io/kubernetes/cmd/kubeadm/app/images.a=' -arc 'k8s.io/kubernetes/cmd/kubeadm/app/phases/certs=k8s.io/kubernetes/cmd/kubeadm/app/phases/certs=bazel-out/k8-fastbuild/bin/cmd/kubeadm/app/phases/certs/linux_amd64_stripped/go_default_library%/k8s.io/kubernetes/cmd/kubeadm/app/phases/certs.a=' -arc 'k8s.io/kubernetes/cmd/kubeadm/app/util=k8s.io/kubernetes/cmd/kubeadm/app/util=bazel-out/k8-fastbuild/bin/cmd/kubeadm/app/util/linux_amd64_stripped/go_default_library%/k8s.io/kubernetes/cmd/kubeadm/app/util.a=' -arc 'k8s.io/kubernetes/cmd/kubeadm/app/util/staticpod=k8s.io/kubernetes/cmd/kubeadm/app/util/staticpod=bazel-out/k8-fastbuild/bin/cmd/kubeadm/app/util/staticpod/linux_amd64_stripped/go_default_library%/k8s.io/kubernetes/cmd/kubeadm/app/util/staticpod.a=' -arc 'k8s.io/kubernetes/pkg/kubeapiserver/authorizer/modes=k8s.io/kubernetes/pkg/kubeapiserver/authorizer/modes=bazel-out/k8-fastbuild/bin/pkg/kubeapiserver/authorizer/modes/linux_amd64_stripped/go_default_library%/k8s.io/kubernetes/pkg/kubeapiserver/authorizer/modes.a=' -arc 'k8s.io/api/core/v1=k8s.io/kubernetes/vendor/k8s.io/api/core/v1=bazel-out/k8-fastbuild/bin/staging/src/k8s.io/api/core/v1/linux_amd64_stripped/go_default_library%/k8s.io/kubernetes/vendor/k8s.io/api/core/v1.a=' -arc 'k8s.io/apimachinery/pkg/util/intstr=k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/intstr=bazel-out/k8-fastbuild/bin/staging/src/k8s.io/apimachinery/pkg/util/intstr/linux_amd64_stripped/go_default_library%/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/intstr.a=' -arc 'k8s.io/apimachinery/pkg/util/sets=k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/sets=bazel-out/k8-fastbuild/bin/staging/src/k8s.io/apimachinery/pkg/util/sets/linux_amd64_stripped/go_default_library%/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/sets.a=' -arc 'k8s.io/apimachinery/pkg/util/version=k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/version=bazel-out/k8-fastbuild/bin/staging/src/k8s.io/apimachinery/pkg/util/version/linux_amd64_stripped/go_default_library%/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/version.a=' -arc 'github.com/pkg/errors=k8s.io/kubernetes/vendor/github.com/pkg/errors=bazel-out/k8-fastbuild/bin/vendor/github.com/pkg/errors/linux_amd64_stripped/go_default_library%/k8s.io/kubernetes/vendor/github.com/pkg/errors.a=' -arc 'k8s.io/klog=k8s.io/kubernetes/vendor/k8s.io/klog=bazel-out/k8-fastbuild/bin/vendor/k8s.io/klog/linux_amd64_stripped/go_default_library%/k8s.io/kubernetes/vendor/k8s.io/klog.a=' -arc 'k8s.io/utils/net=k8s.io/kubernetes/vendor/k8s.io/utils/net=bazel-out/k8-fastbuild/bin/vendor/k8s.io/utils/net/linux_amd64_stripped/go_default_library%/k8s.io/kubernetes/vendor/k8s.io/utils/net.a=' -o bazel-out/k8-fastbuild/bin/cmd/kubeadm/app/phases/controlplane/linux_amd64_stripped/go_default_library%/k8s.io/kubernetes/cmd/kubeadm/app/phases/controlplane.a -package_list bazel-out/host/bin/external/go_sdk/packages.txt -p k8s.io/kubernetes/cmd/kubeadm/app/phases/controlplane -- -trimpath .)
W0712 00:51:34.860] Execution platform: @bazel_tools//platforms:host_platform
W0712 00:51:34.860] 
W0712 00:51:34.893] Use --sandbox_debug to see verbose messages from the sandbox: compile failed: error executing command 
W0712 00:51:34.902]   (cd /bazel-scratch/.cache/bazel/_bazel_prow/48d5366022b4e3197674c8d6e2bee219/sandbox/processwrapper-sandbox/4769/execroot/io_k8s_kubernetes && \
W0712 00:51:34.905]   exec env - \
W0712 00:51:34.910]     CGO_ENABLED=1 \
W0712 00:51:34.916]     GOARCH=amd64 \
W0712 00:51:34.931]     GOOS=linux \
W0712 00:51:34.932]     GOROOT=external/go_sdk \
W0712 00:51:34.942]     GOROOT_FINAL=GOROOT \
W0712 00:51:34.946]     PATH=/usr/bin:/bin \
W0712 00:51:34.950]   bazel-out/host/bin/external/io_bazel_rules_go/go/tools/builders/linux_amd64_stripped/compile -sdk external/go_sdk -installsuffix linux_amd64 -src cmd/kubeadm/app/phases/controlplane/manifests.go -src cmd/kubeadm/app/phases/controlplane/volumes.go -arc 'k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm=k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm=bazel-out/k8-fastbuild/bin/cmd/kubeadm/app/apis/kubeadm/linux_amd64_stripped/go_default_library%/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm.a=' -arc 'k8s.io/kubernetes/cmd/kubeadm/app/constants=k8s.io/kubernetes/cmd/kubeadm/app/constants=bazel-out/k8-fastbuild/bin/cmd/kubeadm/app/constants/linux_amd64_stripped/go_default_library%/k8s.io/kubernetes/cmd/kubeadm/app/constants.a=' -arc 'k8s.io/kubernetes/cmd/kubeadm/app/images=k8s.io/kubernetes/cmd/kubeadm/app/images=bazel-out/k8-fastbuild/bin/cmd/kubeadm/app/images/linux_amd64_stripped/go_default_library%/k8s.io/kubernetes/cmd/kubeadm/app/images.a=' -arc 'k8s.io/kubernetes/cmd/kubeadm/app/phases/certs=k8s.io/kubernetes/cmd/kubeadm/app/phases/certs=bazel-out/k8-fastbuild/bin/cmd/kubeadm/app/phases/certs/linux_amd64_stripped/go_default_library%/k8s.io/kubernetes/cmd/kubeadm/app/phases/certs.a=' -arc 'k8s.io/kubernetes/cmd/kubeadm/app/util=k8s.io/kubernetes/cmd/kubeadm/app/util=bazel-out/k8-fastbuild/bin/cmd/kubeadm/app/util/linux_amd64_stripped/go_default_library%/k8s.io/kubernetes/cmd/kubeadm/app/util.a=' -arc 'k8s.io/kubernetes/cmd/kubeadm/app/util/staticpod=k8s.io/kubernetes/cmd/kubeadm/app/util/staticpod=bazel-out/k8-fastbuild/bin/cmd/kubeadm/app/util/staticpod/linux_amd64_stripped/go_default_library%/k8s.io/kubernetes/cmd/kubeadm/app/util/staticpod.a=' -arc 'k8s.io/kubernetes/pkg/kubeapiserver/authorizer/modes=k8s.io/kubernetes/pkg/kubeapiserver/authorizer/modes=bazel-out/k8-fastbuild/bin/pkg/kubeapiserver/authorizer/modes/linux_amd64_stripped/go_default_library%/k8s.io/kubernetes/pkg/kubeapiserver/authorizer/modes.a=' -arc 'k8s.io/api/core/v1=k8s.io/kubernetes/vendor/k8s.io/api/core/v1=bazel-out/k8-fastbuild/bin/staging/src/k8s.io/api/core/v1/linux_amd64_stripped/go_default_library%/k8s.io/kubernetes/vendor/k8s.io/api/core/v1.a=' -arc 'k8s.io/apimachinery/pkg/util/intstr=k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/intstr=bazel-out/k8-fastbuild/bin/staging/src/k8s.io/apimachinery/pkg/util/intstr/linux_amd64_stripped/go_default_library%/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/intstr.a=' -arc 'k8s.io/apimachinery/pkg/util/sets=k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/sets=bazel-out/k8-fastbuild/bin/staging/src/k8s.io/apimachinery/pkg/util/sets/linux_amd64_stripped/go_default_library%/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/sets.a=' -arc 'k8s.io/apimachinery/pkg/util/version=k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/version=bazel-out/k8-fastbuild/bin/staging/src/k8s.io/apimachinery/pkg/util/version/linux_amd64_stripped/go_default_library%/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/version.a=' -arc 'github.com/pkg/errors=k8s.io/kubernetes/vendor/github.com/pkg/errors=bazel-out/k8-fastbuild/bin/vendor/github.com/pkg/errors/linux_amd64_stripped/go_default_library%/k8s.io/kubernetes/vendor/github.com/pkg/errors.a=' -arc 'k8s.io/klog=k8s.io/kubernetes/vendor/k8s.io/klog=bazel-out/k8-fastbuild/bin/vendor/k8s.io/klog/linux_amd64_stripped/go_default_library%/k8s.io/kubernetes/vendor/k8s.io/klog.a=' -arc 'k8s.io/utils/net=k8s.io/kubernetes/vendor/k8s.io/utils/net=bazel-out/k8-fastbuild/bin/vendor/k8s.io/utils/net/linux_amd64_stripped/go_default_library%/k8s.io/kubernetes/vendor/k8s.io/utils/net.a=' -o bazel-out/k8-fastbuild/bin/cmd/kubeadm/app/phases/controlplane/linux_amd64_stripped/go_default_library%/k8s.io/kubernetes/cmd/kubeadm/app/phases/controlplane.a -package_list bazel-out/host/bin/external/go_sdk/packages.txt -p k8s.io/kubernetes/cmd/kubeadm/app/phases/controlplane -- -trimpath .)
W0712 00:51:34.950] Execution platform: @bazel_tools//platforms:host_platform
W0712 00:51:34.951] 
W0712 00:51:34.951] Use --sandbox_debug to see verbose messages from the sandbox
W0712 00:51:34.951] GoCompile: error running compiler: exit status 2
I0712 00:51:35.052] /bazel-scratch/.cache/bazel/_bazel_prow/48d5366022b4e3197674c8d6e2bee219/sandbox/processwrapper-sandbox/4769/execroot/io_k8s_kubernetes/cmd/kubeadm/app/phases/controlplane/volumes.go:60:31: cannot use cfg (type *kubeadm.ClusterConfiguration) as type *kubeadm.InitConfiguration in argument to mountCertificatesDirectories
I0712 00:51:35.053] /bazel-scratch/.cache/bazel/_bazel_prow/48d5366022b4e3197674c8d6e2bee219/sandbox/processwrapper-sandbox/4769/execroot/io_k8s_kubernetes/cmd/kubeadm/app/phases/controlplane/volumes.go:69:31: cannot use cfg (type *kubeadm.ClusterConfiguration) as type *kubeadm.InitConfiguration in argument to mountCertificatesDirectories
I0712 00:51:35.053] /bazel-scratch/.cache/bazel/_bazel_prow/48d5366022b4e3197674c8d6e2bee219/sandbox/processwrapper-sandbox/4769/execroot/io_k8s_kubernetes/cmd/kubeadm/app/phases/controlplane/volumes.go:80:31: cannot use cfg (type *kubeadm.ClusterConfiguration) as type *kubeadm.InitConfiguration in argument to mountCertificatesDirectories
W0712 00:51:37.333] Target //build/release-tars:release-tars failed to build
W0712 00:51:37.390] INFO: Elapsed time: 463.758s, Critical Path: 124.08s
W0712 00:51:37.392] INFO: 4762 processes: 4762 processwrapper-sandbox.
W0712 00:51:37.401] FAILED: Build did NOT complete successfully
W0712 00:51:37.408] FAILED: Build did NOT complete successfully
W0712 00:51:37.421] make: *** [bazel-release] Error 1
W0712 00:51:37.422] 2019/07/12 00:51:37 process.go:155: Step 'make -C /go/src/k8s.io/kubernetes bazel-release' finished in 7m43.952221681s
W0712 00:51:37.423] 2019/07/12 00:51:37 util.go:255: Flushing memory.
I0712 00:51:37.523] Makefile:638: recipe for target 'bazel-release' failed
I0712 00:51:37.524] make: Leaving directory '/go/src/k8s.io/kubernetes'
W0712 00:51:45.403] 2019/07/12 00:51:45 util.go:265: flushMem error (page cache): exit status 1
W0712 00:51:45.406] 2019/07/12 00:51:45 process.go:96: Saved XML output to /workspace/_artifacts/junit_runner.xml.
W0712 00:51:45.406] 2019/07/12 00:51:45 process.go:153: Running: bash -c . hack/lib/version.sh && KUBE_ROOT=. kube::version::get_version_vars && echo "${KUBE_GIT_VERSION-}"
W0712 00:51:45.907] 2019/07/12 00:51:45 process.go:155: Step 'bash -c . hack/lib/version.sh && KUBE_ROOT=. kube::version::get_version_vars && echo "${KUBE_GIT_VERSION-}"' finished in 500.380179ms
W0712 00:51:45.923] 2019/07/12 00:51:45 main.go:316: Something went wrong: failed to acquire k8s binaries: error during make -C /go/src/k8s.io/kubernetes bazel-release: exit status 2
W0712 00:51:45.923] Traceback (most recent call last):
W0712 00:51:45.924]   File "/workspace/./test-infra/jenkins/../scenarios/kubernetes_e2e.py", line 778, in <module>
W0712 00:51:45.924]     main(parse_args())
W0712 00:51:45.924]   File "/workspace/./test-infra/jenkins/../scenarios/kubernetes_e2e.py", line 626, in main
W0712 00:51:45.924]     mode.start(runner_args)
W0712 00:51:45.924]   File "/workspace/./test-infra/jenkins/../scenarios/kubernetes_e2e.py", line 262, in start
W0712 00:51:45.924]     check_env(env, self.command, *args)
W0712 00:51:45.924]   File "/workspace/./test-infra/jenkins/../scenarios/kubernetes_e2e.py", line 111, in check_env
W0712 00:51:45.925]     subprocess.check_call(cmd, env=env)
W0712 00:51:45.925]   File "/usr/lib/python2.7/subprocess.py", line 186, in check_call
W0712 00:51:45.925]     raise CalledProcessError(retcode, cmd)
W0712 00:51:45.926] subprocess.CalledProcessError: Command '('kubetest', '--dump=/workspace/_artifacts', '--gcp-service-account=/etc/service-account/service-account.json', '--build=bazel', '--stage=gs://kubernetes-release-pull/ci/pull-kubernetes-e2e-gce-100-performance', '--up', '--down', '--provider=gce', '--cluster=e2e-80064-95a39', '--gcp-network=e2e-80064-95a39', '--extract=local', '--gcp-nodes=100', '--gcp-project=k8s-presubmit-scale', '--gcp-zone=us-east1-b', '--test-cmd=/go/src/k8s.io/perf-tests/run-e2e.sh', '--test-cmd-args=cluster-loader2', '--test-cmd-args=--nodes=100', '--test-cmd-args=--provider=gce', '--test-cmd-args=--report-dir=/workspace/_artifacts', '--test-cmd-args=--testconfig=testing/density/config.yaml', '--test-cmd-args=--testconfig=testing/load/config.yaml', '--test-cmd-args=--testoverrides=./testing/density/100_nodes/override.yaml', '--test-cmd-args=--testoverrides=./testing/load/gce/throughput_override.yaml', '--test-cmd-args=--testoverrides=./testing/prometheus/scrape-etcd.yaml', '--test-cmd-args=--testoverrides=./testing/prometheus/scrape-kube-proxy.yaml', '--test-cmd-name=ClusterLoaderV2', '--timeout=100m', '--logexporter-gcs-path=gs://kubernetes-jenkins/pr-logs/pull/80064/pull-kubernetes-e2e-gce-100-performance/1149479051035938816/artifacts')' returned non-zero exit status 1
E0712 00:51:45.928] Command failed
I0712 00:51:45.929] process 523 exited with code 1 after 7.9m
E0712 00:51:45.929] FAIL: pull-kubernetes-e2e-gce-100-performance
I0712 00:51:45.929] Call:  gcloud auth activate-service-account --key-file=/etc/service-account/service-account.json
W0712 00:51:46.685] Activated service account credentials for: [pr-kubekins@kubernetes-jenkins-pull.iam.gserviceaccount.com]
I0712 00:51:46.757] process 60010 exited with code 0 after 0.0m
I0712 00:51:46.757] Call:  gcloud config get-value account
I0712 00:51:47.324] process 60022 exited with code 0 after 0.0m
I0712 00:51:47.325] Will upload results to gs://kubernetes-jenkins/pr-logs using pr-kubekins@kubernetes-jenkins-pull.iam.gserviceaccount.com
I0712 00:51:47.325] Upload result and artifacts...
I0712 00:51:47.325] Gubernator results at https://gubernator.k8s.io/build/kubernetes-jenkins/pr-logs/pull/80064/pull-kubernetes-e2e-gce-100-performance/1149479051035938816
I0712 00:51:47.325] Call:  gsutil ls gs://kubernetes-jenkins/pr-logs/pull/80064/pull-kubernetes-e2e-gce-100-performance/1149479051035938816/artifacts
W0712 00:51:49.212] CommandException: One or more URLs matched no objects.
E0712 00:51:49.433] Command failed
I0712 00:51:49.434] process 60034 exited with code 1 after 0.0m
W0712 00:51:49.434] Remote dir gs://kubernetes-jenkins/pr-logs/pull/80064/pull-kubernetes-e2e-gce-100-performance/1149479051035938816/artifacts not exist yet
I0712 00:51:49.435] Call:  gsutil -m -q -o GSUtil:use_magicfile=True cp -r -c -z log,txt,xml /workspace/_artifacts gs://kubernetes-jenkins/pr-logs/pull/80064/pull-kubernetes-e2e-gce-100-performance/1149479051035938816/artifacts
I0712 00:51:52.196] process 60176 exited with code 0 after 0.0m
I0712 00:51:52.198] Call:  git rev-parse HEAD
I0712 00:51:52.207] process 60700 exited with code 0 after 0.0m
... skipping 21 lines ...