This job view page is being replaced by Spyglass soon. Check out the new job view.
ResultFAILURE
Tests 0 failed / 0 succeeded
Started2020-07-01 11:33
Elapsed1h12m
Revision1967d25c27f6201849dc3941fe5d393ac5f6e09c
links{u'resultstore': {u'url': u'https://source.cloud.google.com/results/invocations/d8205ac8-14d0-4c7c-86ef-8e2ee2b9a582/targets/test'}}
resultstorehttps://source.cloud.google.com/results/invocations/d8205ac8-14d0-4c7c-86ef-8e2ee2b9a582/targets/test

No Test Failures!


Error lines from build-log.txt

... skipping 959 lines ...
NAME               LOCATION    MASTER_VERSION  MASTER_IP     MACHINE_TYPE   NODE_VERSION    NUM_NODES  STATUS
e2e-1967d25-12980  us-east1-b  1.14.10-gke.42  104.196.68.0  e2-standard-8  1.14.10-gke.42  2          RUNNING
++ gcloud container clusters get-credentials e2e-1967d25-12980
Fetching cluster endpoint and auth data.
kubeconfig entry generated for e2e-1967d25-12980.
++ kubectl delete namespace kubeflow --wait
Error from server (NotFound): namespaces "kubeflow" not found
++ echo 'No need to delete kubeflow namespace. It doesn'\''t exist.'
No need to delete kubeflow namespace. It doesn't exist.
++ kubectl create namespace kubeflow --dry-run -o yaml
++ kubectl apply -f -
W0701 12:13:18.017511    1776 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
namespace/kubeflow created
... skipping 530 lines ...
Step #1 - "verify":     title: Database password (Managed storage only)
Step #1 - "verify":     x-google-marketplace:
Step #1 - "verify":       type: MASKED_FIELD
Step #1 - "verify":     description: |-
Step #1 - "verify":       The database password to use when connecting to the Cloud SQL instance.
Step #1 - "verify":       If you leave this field empty, the deployment will try to connect to the instance without providing a password.
Step #1 - "verify":       This will fail if a password is required for the username you provided.
Step #1 - "verify":   managedstorage.databaseNamePrefix:
Step #1 - "verify":     type: string
Step #1 - "verify":     title: Database name prefix (Managed storage only)
Step #1 - "verify":     description: |-
Step #1 - "verify":       The prefix of the database name. Kubeflow Pipelines will create two databases,
Step #1 - "verify":       [prefix]_pipeline and [prefix]_metadata.
... skipping 597 lines ...
Step #1 - "verify": Using /opt/kubectl/1.14/kubectl (server=1.14)
Step #1 - "verify": Using /opt/kubectl/1.14/kubectl (server=1.14)
Step #1 - "verify": Using /opt/kubectl/1.14/kubectl (server=1.14)
Step #1 - "verify": Using /opt/kubectl/1.14/kubectl (server=1.14)
Step #1 - "verify": Using /opt/kubectl/1.14/kubectl (server=1.14)
Step #1 - "verify": Using /opt/kubectl/1.14/kubectl (server=1.14)
Step #1 - "verify": Elapsed 0s
Elapsed 5s
Elapsed 9s
Elapsed 14s
Elapsed 18s
Elapsed 22s
Elapsed 27s
Elapsed 31s
Elapsed 35s
Elapsed 40s
Elapsed 44s
Elapsed 48s
Elapsed 53s
Elapsed 57s
Elapsed 62s
Elapsed 66s
Elapsed 70s
Elapsed 75s
Elapsed 79s
Elapsed 83s
Elapsed 88s
Elapsed 92s
Elapsed 96s
Elapsed 101s
Elapsed 105s
Elapsed 110s
Elapsed 114s
Elapsed 118s
Elapsed 123s
Elapsed 127s
Elapsed 131s
Elapsed 136s
Elapsed 140s
Elapsed 145s
Elapsed 149s
Elapsed 153s
Elapsed 158s
Elapsed 162s
Elapsed 166s
Elapsed 171s
Elapsed 175s
Elapsed 180s
Elapsed 184s
Elapsed 188s
Elapsed 193s
Elapsed 197s
Elapsed 201s
Elapsed 206s
Elapsed 210s
Elapsed 214s
Elapsed 219s
Elapsed 223s
Elapsed 228s
Elapsed 232s
Elapsed 236s
Elapsed 241s
Elapsed 245s
Elapsed 249s
Elapsed 254s
Elapsed 258s
Elapsed 263s
Elapsed 267s
Elapsed 271s
Elapsed 276s
Elapsed 280s
Elapsed 284s
Elapsed 289s
Elapsed 293s
Elapsed 297s
Elapsed 302s
Elapsed 306s
Elapsed 311s
Elapsed 315s
Elapsed 319s
Elapsed 324s
Elapsed 328s
Elapsed 332s
Elapsed 337s
Elapsed 341s
Elapsed 346s
Elapsed 350s
Elapsed 354s
Elapsed 359s
Elapsed 363s
Elapsed 367s
Elapsed 372s
Elapsed 376s
Elapsed 380s
Elapsed 385s
Elapsed 389s
Elapsed 394s
Elapsed 398s
Elapsed 402s
Elapsed 407s
Elapsed 411s
Elapsed 415s
Elapsed 420s
Elapsed 424s
Elapsed 428s
Elapsed 433s
Elapsed 437s
Elapsed 441s
Elapsed 446s
Elapsed 450s
Elapsed 455s
Elapsed 459s
Elapsed 463s
Elapsed 468s
Elapsed 472s
Elapsed 476s
Elapsed 481s
Elapsed 485s
Elapsed 489s
Elapsed 494s
Elapsed 498s
Elapsed 503s
Elapsed 507s
Elapsed 511s
Elapsed 516s
Elapsed 520s
Elapsed 524s
Elapsed 529s
Elapsed 533s
Elapsed 537s
Elapsed 542s
Elapsed 546s
Elapsed 551s
Elapsed 555s
Elapsed 559s
Elapsed 564s
Elapsed 568s
Elapsed 572s
Elapsed 577s
Elapsed 581s
Elapsed 586s
Elapsed 590s
Elapsed 594s
Elapsed 599s
Elapsed 603s
Elapsed 608s
Elapsed 612s
Elapsed 616s
Elapsed 621s
Elapsed 625s
Elapsed 629s
Elapsed 634s
Elapsed 638s
Elapsed 642s
Elapsed 647s
Elapsed 651s
Elapsed 656s
Elapsed 660s
Elapsed 664s
Elapsed 669s
Elapsed 673s
Elapsed 677s
Elapsed 682s
Elapsed 686s
Elapsed 691s
Elapsed 695s
Elapsed 699s
Elapsed 704s
Elapsed 708s
Elapsed 712s
Elapsed 717s
Elapsed 721s
Elapsed 726s
Elapsed 730s
Elapsed 734s
Elapsed 739s
Elapsed 743s
Elapsed 747s
Elapsed 752s
Elapsed 756s
Elapsed 760s
Elapsed 765s
Elapsed 769s
Elapsed 774s
Elapsed 778s
Elapsed 782s
Elapsed 787s
Elapsed 791s
Elapsed 795s
Elapsed 800s
Elapsed 804s
Elapsed 809s
Elapsed 813s
Elapsed 817s
Elapsed 822s
Elapsed 826s
Elapsed 830s
Elapsed 835s
Elapsed 839s
Elapsed 844s
Elapsed 848s
Elapsed 852s
Elapsed 857s
Elapsed 861s
Elapsed 865s
Elapsed 870s
Elapsed 874s
Elapsed 878s
Elapsed 883s
Elapsed 887s
Elapsed 892s
Elapsed 896s
Elapsed 900s
Elapsed 905s
Elapsed 909s
Elapsed 913s
Elapsed 918s
Elapsed 922s
Elapsed 927s
Elapsed 931s
Elapsed 935s
Elapsed 940s
Elapsed 944s
Elapsed 948s
Elapsed 953s
Elapsed 957s
Elapsed 961s
Elapsed 966s
Elapsed 970s
Elapsed 975s
Elapsed 979s
Elapsed 983s
Elapsed 988s
Elapsed 992s
Elapsed 996s
Elapsed 1001s
Elapsed 1005s
Elapsed 1009s
Elapsed 1014s
Elapsed 1018s
Elapsed 1022s
Elapsed 1027s
Elapsed 1031s
Elapsed 1036s
Elapsed 1040s
Elapsed 1044s
Elapsed 1049s
Elapsed 1053s
Elapsed 1057s
Elapsed 1062s
Elapsed 1066s
Elapsed 1070s
Elapsed 1075s
Elapsed 1079s
Elapsed 1084s
Elapsed 1088s
Elapsed 1092s
Elapsed 1097s
Elapsed 1101s
Elapsed 1105s
Elapsed 1110s
Elapsed 1114s
Elapsed 1118s
Elapsed 1123s
Elapsed 1127s
Elapsed 1132s
Elapsed 1136s
Elapsed 1140s
Elapsed 1145s
Elapsed 1149s
Elapsed 1153s
Elapsed 1158s
Elapsed 1162s
Elapsed 1167s
Elapsed 1171s
Elapsed 1175s
Elapsed 1180s
Elapsed 1184s
Elapsed 1188s
Elapsed 1193s
Elapsed 1197s
Elapsed 1202s
Elapsed 1206s
Elapsed 1210s
Elapsed 1215s
Elapsed 1219s
Elapsed 1223s
Elapsed 1228s
Elapsed 1232s
Elapsed 1237s
Elapsed 1241s
Elapsed 1245s
Elapsed 1250s
Elapsed 1254s
Elapsed 1258s
Elapsed 1263s
Elapsed 1267s
Elapsed 1271s
Elapsed 1276s
Elapsed 1280s
Elapsed 1284s
Elapsed 1289s
Elapsed 1293s
Elapsed 1298s
Elapsed 1302s
Elapsed 1306s
Elapsed 1311s
Elapsed 1315s
Elapsed 1319s
Elapsed 1324s
Elapsed 1328s
Elapsed 1332s
Elapsed 1337s
Elapsed 1341s
Elapsed 1346s
Elapsed 1350s
Elapsed 1354s
Elapsed 1359s
Elapsed 1363s
Elapsed 1367s
Elapsed 1372s
Elapsed 1376s
Elapsed 1380s
Elapsed 1385s
Elapsed 1389s
Elapsed 1394s
Elapsed 1398s
Elapsed 1402s
Elapsed 1407s
Elapsed 1411s
Elapsed 1415s
Elapsed 1420s
Elapsed 1424s
Elapsed 1428s
Elapsed 1433s
Elapsed 1437s
Elapsed 1442s
Elapsed 1446s
Elapsed 1450s
Elapsed 1455s
Elapsed 1459s
Elapsed 1463s
Elapsed 1468s
Elapsed 1472s
Elapsed 1476s
Elapsed 1481s
Elapsed 1485s
Elapsed 1489s
Elapsed 1494s
Elapsed 1498s
Elapsed 1503s
Elapsed 1507s
Elapsed 1511s
Elapsed 1516s
Elapsed 1520s
Elapsed 1524s
Elapsed 1529s
Elapsed 1533s
Elapsed 1537s
Elapsed 1542s
Elapsed 1546s
Elapsed 1551s
Elapsed 1555s
Elapsed 1559s
Elapsed 1564s
Elapsed 1568s
Elapsed 1572s
Elapsed 1577s
Elapsed 1581s
Elapsed 1585s
Elapsed 1590s
Elapsed 1594s
Elapsed 1599s
Elapsed 1603s
Elapsed 1607s
Elapsed 1612s
Elapsed 1616s
Elapsed 1620s
Elapsed 1625s
Elapsed 1629s
Elapsed 1634s
Elapsed 1638s
Elapsed 1642s
Elapsed 1647s
Elapsed 1651s
Elapsed 1655s
Elapsed 1660s
Elapsed 1664s
Elapsed 1668s
Elapsed 1673s
Elapsed 1677s
Elapsed 1681s
Elapsed 1686s
Elapsed 1690s
Elapsed 1695s
Elapsed 1699s
Elapsed 1703s
Elapsed 1708s
Elapsed 1712s
Elapsed 1716s
Elapsed 1721s
Elapsed 1725s
Elapsed 1729s
Elapsed 1734s
Elapsed 1738s
Elapsed 1743s
Elapsed 1747s
Elapsed 1751s
Elapsed 1756s
Elapsed 1760s
Elapsed 1764s
Elapsed 1769s
Elapsed 1773s
Elapsed 1777s
Elapsed 1782s
Elapsed 1786s
Elapsed 1791s
Elapsed 1795s
Elapsed 1799s
Elapsed 1804s
ERROR Deployer job timeout
Step #1 - "verify": INFO cleaning all resources.
Step #1 - "verify": INFO Collecting events for namespace "apptest-fkf3mzk6"
Step #1 - "verify": Using /opt/kubectl/1.14/kubectl (server=1.14)
Step #1 - "verify": LAST SEEN   TYPE      REASON                   OBJECT                                                  MESSAGE
Step #1 - "verify": 30m         Normal    Scheduled                pod/apptest-fkf3mzk6-deployer-nndrc                     Successfully assigned apptest-fkf3mzk6/apptest-fkf3mzk6-deployer-nndrc to gke-hosted-verify-test-default-pool-53177527-rcfl
Step #1 - "verify": 30m         Normal    Pulling                  pod/apptest-fkf3mzk6-deployer-nndrc                     Pulling image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/deployer:1.0.0-dev.2"
Step #1 - "verify": 29m         Normal    Pulled                   pod/apptest-fkf3mzk6-deployer-nndrc                     Successfully pulled image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/deployer:1.0.0-dev.2"
Step #1 - "verify": 29m         Normal    Created                  pod/apptest-fkf3mzk6-deployer-nndrc                     Created container deployer
Step #1 - "verify": 29m         Normal    Started                  pod/apptest-fkf3mzk6-deployer-nndrc                     Started container deployer
Step #1 - "verify": 30m         Normal    SuccessfulCreate         job/apptest-fkf3mzk6-deployer                           Created pod: apptest-fkf3mzk6-deployer-nndrc
Step #1 - "verify": 29m         Normal    Scheduled                pod/cache-deployer-deployment-784cfb566c-9p9v6          Successfully assigned apptest-fkf3mzk6/cache-deployer-deployment-784cfb566c-9p9v6 to gke-hosted-verify-test-default-pool-53177527-rcfl
Step #1 - "verify": 27m         Normal    Pulling                  pod/cache-deployer-deployment-784cfb566c-9p9v6          Pulling image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/cachedeployer:0.5.1"
Step #1 - "verify": 27m         Warning   Failed                   pod/cache-deployer-deployment-784cfb566c-9p9v6          Failed to pull image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/cachedeployer:0.5.1": rpc error: code = Unknown desc = Error response from daemon: manifest for gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/cachedeployer:0.5.1 not found
Step #1 - "verify": 27m         Warning   Failed                   pod/cache-deployer-deployment-784cfb566c-9p9v6          Error: ErrImagePull
Step #1 - "verify": 27m         Normal    BackOff                  pod/cache-deployer-deployment-784cfb566c-9p9v6          Back-off pulling image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/cachedeployer:0.5.1"
Step #1 - "verify": 4m16s       Warning   Failed                   pod/cache-deployer-deployment-784cfb566c-9p9v6          Error: ImagePullBackOff
Step #1 - "verify": 29m         Normal    SuccessfulCreate         replicaset/cache-deployer-deployment-784cfb566c         Created pod: cache-deployer-deployment-784cfb566c-9p9v6
Step #1 - "verify": 29m         Normal    ScalingReplicaSet        deployment/cache-deployer-deployment                    Scaled up replica set cache-deployer-deployment-784cfb566c to 1
Step #1 - "verify": 29m         Normal    Scheduled                pod/cache-server-656db897-khfj6                         Successfully assigned apptest-fkf3mzk6/cache-server-656db897-khfj6 to gke-hosted-verify-test-default-pool-53177527-dkq2
Step #1 - "verify": 2m49s       Warning   FailedMount              pod/cache-server-656db897-khfj6                         MountVolume.SetUp failed for volume "webhook-tls-certs" : secret "webhook-server-tls" not found
Step #1 - "verify": 6m53s       Warning   FailedMount              pod/cache-server-656db897-khfj6                         Unable to mount volumes for pod "cache-server-656db897-khfj6_apptest-fkf3mzk6(a4b5821a-bb94-11ea-b70f-42010a800202)": timeout expired waiting for volumes to attach or mount for pod "apptest-fkf3mzk6"/"cache-server-656db897-khfj6". list of unmounted volumes=[webhook-tls-certs]. list of unattached volumes=[webhook-tls-certs kubeflow-pipelines-cache-token-gtkxn]
Step #1 - "verify": 29m         Normal    SuccessfulCreate         replicaset/cache-server-656db897                        Created pod: cache-server-656db897-khfj6
Step #1 - "verify": 29m         Normal    ScalingReplicaSet        deployment/cache-server                                 Scaled up replica set cache-server-656db897 to 1
Step #1 - "verify": 29m         Normal    Scheduled                pod/metadata-envoy-79fc9d769-57s42                      Successfully assigned apptest-fkf3mzk6/metadata-envoy-79fc9d769-57s42 to gke-hosted-verify-test-default-pool-53177527-7rzm
Step #1 - "verify": 27m         Normal    Pulling                  pod/metadata-envoy-79fc9d769-57s42                      Pulling image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/metadataenvoy:0.5.1"
Step #1 - "verify": 27m         Warning   Failed                   pod/metadata-envoy-79fc9d769-57s42                      Failed to pull image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/metadataenvoy:0.5.1": rpc error: code = Unknown desc = Error response from daemon: manifest for gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/metadataenvoy:0.5.1 not found
Step #1 - "verify": 27m         Warning   Failed                   pod/metadata-envoy-79fc9d769-57s42                      Error: ErrImagePull
Step #1 - "verify": 27m         Normal    BackOff                  pod/metadata-envoy-79fc9d769-57s42                      Back-off pulling image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/metadataenvoy:0.5.1"
Step #1 - "verify": 4m11s       Warning   Failed                   pod/metadata-envoy-79fc9d769-57s42                      Error: ImagePullBackOff
Step #1 - "verify": 29m         Normal    SuccessfulCreate         replicaset/metadata-envoy-79fc9d769                     Created pod: metadata-envoy-79fc9d769-57s42
Step #1 - "verify": 29m         Normal    ScalingReplicaSet        deployment/metadata-envoy                               Scaled up replica set metadata-envoy-79fc9d769 to 1
Step #1 - "verify": 29m         Normal    Scheduled                pod/metadata-grpc-deployment-7d75ddb958-gtrt9           Successfully assigned apptest-fkf3mzk6/metadata-grpc-deployment-7d75ddb958-gtrt9 to gke-hosted-verify-test-default-pool-53177527-dkq2
Step #1 - "verify": 27m         Normal    Pulling                  pod/metadata-grpc-deployment-7d75ddb958-gtrt9           Pulling image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/metadataserver:0.5.1"
Step #1 - "verify": 27m         Warning   Failed                   pod/metadata-grpc-deployment-7d75ddb958-gtrt9           Failed to pull image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/metadataserver:0.5.1": rpc error: code = Unknown desc = Error response from daemon: manifest for gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/metadataserver:0.5.1 not found
Step #1 - "verify": 27m         Warning   Failed                   pod/metadata-grpc-deployment-7d75ddb958-gtrt9           Error: ErrImagePull
Step #1 - "verify": 4m16s       Normal    BackOff                  pod/metadata-grpc-deployment-7d75ddb958-gtrt9           Back-off pulling image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/metadataserver:0.5.1"
Step #1 - "verify": 9m17s       Warning   Failed                   pod/metadata-grpc-deployment-7d75ddb958-gtrt9           Error: ImagePullBackOff
Step #1 - "verify": 29m         Normal    SuccessfulCreate         replicaset/metadata-grpc-deployment-7d75ddb958          Created pod: metadata-grpc-deployment-7d75ddb958-gtrt9
Step #1 - "verify": 29m         Normal    ScalingReplicaSet        deployment/metadata-grpc-deployment                     Scaled up replica set metadata-grpc-deployment-7d75ddb958 to 1
Step #1 - "verify": 29m         Normal    Scheduled                pod/metadata-writer-7c85b864fb-2sglk                    Successfully assigned apptest-fkf3mzk6/metadata-writer-7c85b864fb-2sglk to gke-hosted-verify-test-default-pool-53177527-rcfl
Step #1 - "verify": 27m         Normal    Pulling                  pod/metadata-writer-7c85b864fb-2sglk                    Pulling image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/metadatawriter:0.5.1"
Step #1 - "verify": 27m         Warning   Failed                   pod/metadata-writer-7c85b864fb-2sglk                    Failed to pull image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/metadatawriter:0.5.1": rpc error: code = Unknown desc = Error response from daemon: manifest for gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/metadatawriter:0.5.1 not found
Step #1 - "verify": 27m         Warning   Failed                   pod/metadata-writer-7c85b864fb-2sglk                    Error: ErrImagePull
Step #1 - "verify": 4m13s       Normal    BackOff                  pod/metadata-writer-7c85b864fb-2sglk                    Back-off pulling image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/metadatawriter:0.5.1"
Step #1 - "verify": 9m7s        Warning   Failed                   pod/metadata-writer-7c85b864fb-2sglk                    Error: ImagePullBackOff
Step #1 - "verify": 29m         Normal    SuccessfulCreate         replicaset/metadata-writer-7c85b864fb                   Created pod: metadata-writer-7c85b864fb-2sglk
Step #1 - "verify": 29m         Normal    ScalingReplicaSet        deployment/metadata-writer                              Scaled up replica set metadata-writer-7c85b864fb to 1
Step #1 - "verify": 29m         Warning   FailedScheduling         pod/minio-76566f85c6-jdgcz                              pod has unbound immediate PersistentVolumeClaims (repeated 3 times)
Step #1 - "verify": 29m         Normal    Scheduled                pod/minio-76566f85c6-jdgcz                              Successfully assigned apptest-fkf3mzk6/minio-76566f85c6-jdgcz to gke-hosted-verify-test-default-pool-53177527-rcfl
Step #1 - "verify": 29m         Normal    SuccessfulAttachVolume   pod/minio-76566f85c6-jdgcz                              AttachVolume.Attach succeeded for volume "pvc-a5ef683d-bb94-11ea-b70f-42010a800202"
Step #1 - "verify": 27m         Normal    Pulling                  pod/minio-76566f85c6-jdgcz                              Pulling image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/minio:0.5.1"
Step #1 - "verify": 27m         Warning   Failed                   pod/minio-76566f85c6-jdgcz                              Failed to pull image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/minio:0.5.1": rpc error: code = Unknown desc = Error response from daemon: manifest for gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/minio:0.5.1 not found
Step #1 - "verify": 27m         Warning   Failed                   pod/minio-76566f85c6-jdgcz                              Error: ErrImagePull
Step #1 - "verify": 18m         Normal    BackOff                  pod/minio-76566f85c6-jdgcz                              Back-off pulling image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/minio:0.5.1"
Step #1 - "verify": 3m53s       Warning   Failed                   pod/minio-76566f85c6-jdgcz                              Error: ImagePullBackOff
Step #1 - "verify": 29m         Normal    SuccessfulCreate         replicaset/minio-76566f85c6                             Created pod: minio-76566f85c6-jdgcz
Step #1 - "verify": 29m         Normal    ProvisioningSucceeded    persistentvolumeclaim/minio-pvc                         Successfully provisioned volume pvc-a5ef683d-bb94-11ea-b70f-42010a800202 using kubernetes.io/gce-pd
Step #1 - "verify": 29m         Normal    ScalingReplicaSet        deployment/minio                                        Scaled up replica set minio-76566f85c6 to 1
Step #1 - "verify": 29m         Normal    Scheduled                pod/ml-pipeline-59d49c8f68-qrk27                        Successfully assigned apptest-fkf3mzk6/ml-pipeline-59d49c8f68-qrk27 to gke-hosted-verify-test-default-pool-53177527-7rzm
Step #1 - "verify": 27m         Normal    Pulling                  pod/ml-pipeline-59d49c8f68-qrk27                        Pulling image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/apiserver:0.5.1"
Step #1 - "verify": 27m         Warning   Failed                   pod/ml-pipeline-59d49c8f68-qrk27                        Failed to pull image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/apiserver:0.5.1": rpc error: code = Unknown desc = Error response from daemon: manifest for gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/apiserver:0.5.1 not found
Step #1 - "verify": 27m         Warning   Failed                   pod/ml-pipeline-59d49c8f68-qrk27                        Error: ErrImagePull
Step #1 - "verify": 24m         Normal    BackOff                  pod/ml-pipeline-59d49c8f68-qrk27                        Back-off pulling image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/apiserver:0.5.1"
Step #1 - "verify": 4m10s       Warning   Failed                   pod/ml-pipeline-59d49c8f68-qrk27                        Error: ImagePullBackOff
Step #1 - "verify": 29m         Normal    SuccessfulCreate         replicaset/ml-pipeline-59d49c8f68                       Created pod: ml-pipeline-59d49c8f68-qrk27
Step #1 - "verify": 29m         Normal    Scheduled                pod/ml-pipeline-persistenceagent-58b4c68c67-sdghx       Successfully assigned apptest-fkf3mzk6/ml-pipeline-persistenceagent-58b4c68c67-sdghx to gke-hosted-verify-test-default-pool-53177527-rcfl
Step #1 - "verify": 27m         Normal    Pulling                  pod/ml-pipeline-persistenceagent-58b4c68c67-sdghx       Pulling image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/persistenceagent:0.5.1"
Step #1 - "verify": 27m         Warning   Failed                   pod/ml-pipeline-persistenceagent-58b4c68c67-sdghx       Failed to pull image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/persistenceagent:0.5.1": rpc error: code = Unknown desc = Error response from daemon: manifest for gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/persistenceagent:0.5.1 not found
Step #1 - "verify": 27m         Warning   Failed                   pod/ml-pipeline-persistenceagent-58b4c68c67-sdghx       Error: ErrImagePull
Step #1 - "verify": 9m9s        Normal    BackOff                  pod/ml-pipeline-persistenceagent-58b4c68c67-sdghx       Back-off pulling image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/persistenceagent:0.5.1"
Step #1 - "verify": 4m6s        Warning   Failed                   pod/ml-pipeline-persistenceagent-58b4c68c67-sdghx       Error: ImagePullBackOff
Step #1 - "verify": 29m         Normal    SuccessfulCreate         replicaset/ml-pipeline-persistenceagent-58b4c68c67      Created pod: ml-pipeline-persistenceagent-58b4c68c67-sdghx
Step #1 - "verify": 29m         Normal    ScalingReplicaSet        deployment/ml-pipeline-persistenceagent                 Scaled up replica set ml-pipeline-persistenceagent-58b4c68c67 to 1
Step #1 - "verify": 29m         Normal    Scheduled                pod/ml-pipeline-scheduledworkflow-ddc5f6b8c-8snzn       Successfully assigned apptest-fkf3mzk6/ml-pipeline-scheduledworkflow-ddc5f6b8c-8snzn to gke-hosted-verify-test-default-pool-53177527-dkq2
Step #1 - "verify": 27m         Normal    Pulling                  pod/ml-pipeline-scheduledworkflow-ddc5f6b8c-8snzn       Pulling image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/scheduledworkflow:0.5.1"
Step #1 - "verify": 27m         Warning   Failed                   pod/ml-pipeline-scheduledworkflow-ddc5f6b8c-8snzn       Failed to pull image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/scheduledworkflow:0.5.1": rpc error: code = Unknown desc = Error response from daemon: manifest for gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/scheduledworkflow:0.5.1 not found
Step #1 - "verify": 27m         Warning   Failed                   pod/ml-pipeline-scheduledworkflow-ddc5f6b8c-8snzn       Error: ErrImagePull
Step #1 - "verify": 27m         Normal    BackOff                  pod/ml-pipeline-scheduledworkflow-ddc5f6b8c-8snzn       Back-off pulling image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/scheduledworkflow:0.5.1"
Step #1 - "verify": 4m14s       Warning   Failed                   pod/ml-pipeline-scheduledworkflow-ddc5f6b8c-8snzn       Error: ImagePullBackOff
Step #1 - "verify": 29m         Normal    SuccessfulCreate         replicaset/ml-pipeline-scheduledworkflow-ddc5f6b8c      Created pod: ml-pipeline-scheduledworkflow-ddc5f6b8c-8snzn
Step #1 - "verify": 29m         Normal    ScalingReplicaSet        deployment/ml-pipeline-scheduledworkflow                Scaled up replica set ml-pipeline-scheduledworkflow-ddc5f6b8c to 1
Step #1 - "verify": 29m         Normal    Scheduled                pod/ml-pipeline-ui-6cb987c9df-ljqwj                     Successfully assigned apptest-fkf3mzk6/ml-pipeline-ui-6cb987c9df-ljqwj to gke-hosted-verify-test-default-pool-53177527-dkq2
Step #1 - "verify": 27m         Normal    Pulling                  pod/ml-pipeline-ui-6cb987c9df-ljqwj                     Pulling image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/frontend:0.5.1"
Step #1 - "verify": 27m         Warning   Failed                   pod/ml-pipeline-ui-6cb987c9df-ljqwj                     Failed to pull image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/frontend:0.5.1": rpc error: code = Unknown desc = Error response from daemon: manifest for gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/frontend:0.5.1 not found
Step #1 - "verify": 27m         Warning   Failed                   pod/ml-pipeline-ui-6cb987c9df-ljqwj                     Error: ErrImagePull
Step #1 - "verify": 9m8s        Normal    BackOff                  pod/ml-pipeline-ui-6cb987c9df-ljqwj                     Back-off pulling image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/frontend:0.5.1"
Step #1 - "verify": 4m6s        Warning   Failed                   pod/ml-pipeline-ui-6cb987c9df-ljqwj                     Error: ImagePullBackOff
Step #1 - "verify": 29m         Normal    SuccessfulCreate         replicaset/ml-pipeline-ui-6cb987c9df                    Created pod: ml-pipeline-ui-6cb987c9df-ljqwj
Step #1 - "verify": 29m         Normal    ScalingReplicaSet        deployment/ml-pipeline-ui                               Scaled up replica set ml-pipeline-ui-6cb987c9df to 1
Step #1 - "verify": 29m         Normal    Scheduled                pod/ml-pipeline-viewer-crd-58bbff9745-wdthk             Successfully assigned apptest-fkf3mzk6/ml-pipeline-viewer-crd-58bbff9745-wdthk to gke-hosted-verify-test-default-pool-53177527-dkq2
Step #1 - "verify": 27m         Normal    Pulling                  pod/ml-pipeline-viewer-crd-58bbff9745-wdthk             Pulling image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/viewercrd:0.5.1"
Step #1 - "verify": 27m         Warning   Failed                   pod/ml-pipeline-viewer-crd-58bbff9745-wdthk             Failed to pull image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/viewercrd:0.5.1": rpc error: code = Unknown desc = Error response from daemon: manifest for gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/viewercrd:0.5.1 not found
Step #1 - "verify": 27m         Warning   Failed                   pod/ml-pipeline-viewer-crd-58bbff9745-wdthk             Error: ErrImagePull
Step #1 - "verify": 27m         Normal    BackOff                  pod/ml-pipeline-viewer-crd-58bbff9745-wdthk             Back-off pulling image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/viewercrd:0.5.1"
Step #1 - "verify": 4m9s        Warning   Failed                   pod/ml-pipeline-viewer-crd-58bbff9745-wdthk             Error: ImagePullBackOff
Step #1 - "verify": 29m         Normal    SuccessfulCreate         replicaset/ml-pipeline-viewer-crd-58bbff9745            Created pod: ml-pipeline-viewer-crd-58bbff9745-wdthk
Step #1 - "verify": 29m         Normal    ScalingReplicaSet        deployment/ml-pipeline-viewer-crd                       Scaled up replica set ml-pipeline-viewer-crd-58bbff9745 to 1
Step #1 - "verify": 29m         Normal    Scheduled                pod/ml-pipeline-visualizationserver-574774d768-rr8gb    Successfully assigned apptest-fkf3mzk6/ml-pipeline-visualizationserver-574774d768-rr8gb to gke-hosted-verify-test-default-pool-53177527-rcfl
Step #1 - "verify": 27m         Normal    Pulling                  pod/ml-pipeline-visualizationserver-574774d768-rr8gb    Pulling image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/visualizationserver:0.5.1"
Step #1 - "verify": 27m         Warning   Failed                   pod/ml-pipeline-visualizationserver-574774d768-rr8gb    Failed to pull image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/visualizationserver:0.5.1": rpc error: code = Unknown desc = Error response from daemon: manifest for gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/visualizationserver:0.5.1 not found
Step #1 - "verify": 27m         Warning   Failed                   pod/ml-pipeline-visualizationserver-574774d768-rr8gb    Error: ErrImagePull
Step #1 - "verify": 4m6s        Normal    BackOff                  pod/ml-pipeline-visualizationserver-574774d768-rr8gb    Back-off pulling image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/visualizationserver:0.5.1"
Step #1 - "verify": 9m12s       Warning   Failed                   pod/ml-pipeline-visualizationserver-574774d768-rr8gb    Error: ImagePullBackOff
Step #1 - "verify": 29m         Normal    SuccessfulCreate         replicaset/ml-pipeline-visualizationserver-574774d768   Created pod: ml-pipeline-visualizationserver-574774d768-rr8gb
Step #1 - "verify": 29m         Normal    ScalingReplicaSet        deployment/ml-pipeline-visualizationserver              Scaled up replica set ml-pipeline-visualizationserver-574774d768 to 1
Step #1 - "verify": 29m         Normal    ScalingReplicaSet        deployment/ml-pipeline                                  Scaled up replica set ml-pipeline-59d49c8f68 to 1
Step #1 - "verify": 29m         Warning   FailedScheduling         pod/mysql-9cb48b5cc-b8h54                               pod has unbound immediate PersistentVolumeClaims (repeated 3 times)
Step #1 - "verify": 29m         Normal    Scheduled                pod/mysql-9cb48b5cc-b8h54                               Successfully assigned apptest-fkf3mzk6/mysql-9cb48b5cc-b8h54 to gke-hosted-verify-test-default-pool-53177527-rcfl
Step #1 - "verify": 29m         Normal    SuccessfulAttachVolume   pod/mysql-9cb48b5cc-b8h54                               AttachVolume.Attach succeeded for volume "pvc-a5ff66f0-bb94-11ea-b70f-42010a800202"
Step #1 - "verify": 27m         Normal    Pulling                  pod/mysql-9cb48b5cc-b8h54                               Pulling image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/mysql:0.5.1"
Step #1 - "verify": 27m         Warning   Failed                   pod/mysql-9cb48b5cc-b8h54                               Failed to pull image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/mysql:0.5.1": rpc error: code = Unknown desc = Error response from daemon: manifest for gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/mysql:0.5.1 not found
Step #1 - "verify": 27m         Warning   Failed                   pod/mysql-9cb48b5cc-b8h54                               Error: ErrImagePull
Step #1 - "verify": 3m55s       Normal    BackOff                  pod/mysql-9cb48b5cc-b8h54                               Back-off pulling image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/mysql:0.5.1"
Step #1 - "verify": 8m49s       Warning   Failed                   pod/mysql-9cb48b5cc-b8h54                               Error: ImagePullBackOff
Step #1 - "verify": 29m         Normal    SuccessfulCreate         replicaset/mysql-9cb48b5cc                              Created pod: mysql-9cb48b5cc-b8h54
Step #1 - "verify": 29m         Normal    ProvisioningSucceeded    persistentvolumeclaim/mysql-pv-claim                    Successfully provisioned volume pvc-a5ff66f0-bb94-11ea-b70f-42010a800202 using kubernetes.io/gce-pd
Step #1 - "verify": 29m         Normal    ScalingReplicaSet        deployment/mysql                                        Scaled up replica set mysql-9cb48b5cc to 1
Step #1 - "verify": 29m         Normal    Scheduled                pod/proxy-agent-75d9f4659d-l6rb9                        Successfully assigned apptest-fkf3mzk6/proxy-agent-75d9f4659d-l6rb9 to gke-hosted-verify-test-default-pool-53177527-rcfl
Step #1 - "verify": 27m         Normal    Pulling                  pod/proxy-agent-75d9f4659d-l6rb9                        Pulling image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/proxyagent:0.5.1"
Step #1 - "verify": 27m         Warning   Failed                   pod/proxy-agent-75d9f4659d-l6rb9                        Failed to pull image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/proxyagent:0.5.1": rpc error: code = Unknown desc = Error response from daemon: manifest for gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/proxyagent:0.5.1 not found
Step #1 - "verify": 27m         Warning   Failed                   pod/proxy-agent-75d9f4659d-l6rb9                        Error: ErrImagePull
Step #1 - "verify": 19m         Normal    BackOff                  pod/proxy-agent-75d9f4659d-l6rb9                        Back-off pulling image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/proxyagent:0.5.1"
Step #1 - "verify": 4m12s       Warning   Failed                   pod/proxy-agent-75d9f4659d-l6rb9                        Error: ImagePullBackOff
Step #1 - "verify": 29m         Normal    SuccessfulCreate         replicaset/proxy-agent-75d9f4659d                       Created pod: proxy-agent-75d9f4659d-l6rb9
Step #1 - "verify": 29m         Normal    ScalingReplicaSet        deployment/proxy-agent                                  Scaled up replica set proxy-agent-75d9f4659d to 1
Step #1 - "verify": 29m         Normal    Scheduled                pod/workflow-controller-5855876fb9-78h78                Successfully assigned apptest-fkf3mzk6/workflow-controller-5855876fb9-78h78 to gke-hosted-verify-test-default-pool-53177527-rcfl
Step #1 - "verify": 27m         Normal    Pulling                  pod/workflow-controller-5855876fb9-78h78                Pulling image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/argoworkflowcontroller:0.5.1"
Step #1 - "verify": 28m         Warning   Failed                   pod/workflow-controller-5855876fb9-78h78                Failed to pull image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/argoworkflowcontroller:0.5.1": rpc error: code = Unknown desc = Error response from daemon: manifest for gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/argoworkflowcontroller:0.5.1 not found
Step #1 - "verify": 28m         Warning   Failed                   pod/workflow-controller-5855876fb9-78h78                Error: ErrImagePull
Step #1 - "verify": 29m         Normal    SandboxChanged           pod/workflow-controller-5855876fb9-78h78                Pod sandbox changed, it will be killed and re-created.
Step #1 - "verify": 24m         Normal    BackOff                  pod/workflow-controller-5855876fb9-78h78                Back-off pulling image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/argoworkflowcontroller:0.5.1"
Step #1 - "verify": 4m12s       Warning   Failed                   pod/workflow-controller-5855876fb9-78h78                Error: ImagePullBackOff
Step #1 - "verify": 29m         Normal    SuccessfulCreate         replicaset/workflow-controller-5855876fb9               Created pod: workflow-controller-5855876fb9-78h78
Step #1 - "verify": 29m         Normal    ScalingReplicaSet        deployment/workflow-controller                          Scaled up replica set workflow-controller-5855876fb9 to 1
Step #1 - "verify": Using /opt/kubectl/1.14/kubectl (server=1.14)
Step #1 - "verify": INFO Collecting logs for deployer
Step #1 - "verify": Using /opt/kubectl/1.14/kubectl (server=1.14)
Step #1 - "verify":   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Step #1 - "verify":                                  Dload  Upload   Total   Spent    Left  Speed
Step #1 - "verify": 
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100    16  100    16    0     0    232      0 --:--:-- --:--:-- --:--:--   238
Step #1 - "verify": Using /opt/kubectl/1.14/kubectl (server=1.14)
Step #1 - "verify": Error from server (NotFound): configmaps "gcp-default-config" not found
Step #1 - "verify": Using /opt/kubectl/1.14/kubectl (server=1.14)
Step #1 - "verify": configmap/gcp-default-config created
Step #1 - "verify": init_action done
Step #1 - "verify": + trap handle_failure EXIT
Step #1 - "verify": + LOG_SMOKE_TEST=SMOKE_TEST
Step #1 - "verify": + test_schema=/data-test/schema.yaml
... skipping 326 lines ...
Step #1 - "verify": application.app.k8s.io/apptest-fkf3mzk6 configured
Step #1 - "verify": + for i in "$@"
Step #1 - "verify": + case $i in
Step #1 - "verify": + status=Success
Step #1 - "verify": + shift
Step #1 - "verify": + [[ -z Success ]]
Step #1 - "verify": + [[ Success =~ ^(Pending|Success|Failed)$ ]]
Step #1 - "verify": Marking deployment of application "apptest-fkf3mzk6" as "Success".
Step #1 - "verify": + [[ -z apptest-fkf3mzk6 ]]
Step #1 - "verify": + [[ -z apptest-fkf3mzk6 ]]
Step #1 - "verify": + echo 'Marking deployment of application "apptest-fkf3mzk6" as "Success".'
Step #1 - "verify": + [[ Success == \S\u\c\c\e\s\s ]]
Step #1 - "verify": ++ print_published_version.py --empty_if_not_supported
... skipping 5472 lines ...
Step #1 - "verify": kind: List
Step #1 - "verify": metadata:
Step #1 - "verify":   resourceVersion: ""
Step #1 - "verify":   selfLink: ""
Step #1 - "verify": Using /opt/kubectl/1.14/kubectl (server=1.14)
Step #1 - "verify": namespace "apptest-fkf3mzk6" deleted
Step #1 - "verify": === ERROR SUMMARY ===
Step #1 - "verify": ERROR Deployer job timeout
Step #1 - "verify": 
Step #1 - "verify": Error events found in namespace "apptest-fkf3mzk6"
Step #1 - "verify": LAST SEEN   TYPE      REASON             OBJECT                                                 MESSAGE
Step #1 - "verify": 27m         Warning   Failed             pod/cache-deployer-deployment-784cfb566c-9p9v6         Failed to pull image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/cachedeployer:0.5.1": rpc error: code = Unknown desc = Error response from daemon: manifest for gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/cachedeployer:0.5.1 not found
Step #1 - "verify": 27m         Warning   Failed             pod/cache-deployer-deployment-784cfb566c-9p9v6         Error: ErrImagePull
Step #1 - "verify": 4m17s       Warning   Failed             pod/cache-deployer-deployment-784cfb566c-9p9v6         Error: ImagePullBackOff
Step #1 - "verify": 2m50s       Warning   FailedMount        pod/cache-server-656db897-khfj6                        MountVolume.SetUp failed for volume "webhook-tls-certs" : secret "webhook-server-tls" not found
Step #1 - "verify": 6m54s       Warning   FailedMount        pod/cache-server-656db897-khfj6                        Unable to mount volumes for pod "cache-server-656db897-khfj6_apptest-fkf3mzk6(a4b5821a-bb94-11ea-b70f-42010a800202)": timeout expired waiting for volumes to attach or mount for pod "apptest-fkf3mzk6"/"cache-server-656db897-khfj6". list of unmounted volumes=[webhook-tls-certs]. list of unattached volumes=[webhook-tls-certs kubeflow-pipelines-cache-token-gtkxn]
Step #1 - "verify": 27m         Warning   Failed             pod/metadata-envoy-79fc9d769-57s42                     Failed to pull image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/metadataenvoy:0.5.1": rpc error: code = Unknown desc = Error response from daemon: manifest for gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/metadataenvoy:0.5.1 not found
Step #1 - "verify": 27m         Warning   Failed             pod/metadata-envoy-79fc9d769-57s42                     Error: ErrImagePull
Step #1 - "verify": 4m12s       Warning   Failed             pod/metadata-envoy-79fc9d769-57s42                     Error: ImagePullBackOff
Step #1 - "verify": 27m         Warning   Failed             pod/metadata-grpc-deployment-7d75ddb958-gtrt9          Failed to pull image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/metadataserver:0.5.1": rpc error: code = Unknown desc = Error response from daemon: manifest for gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/metadataserver:0.5.1 not found
Step #1 - "verify": 27m         Warning   Failed             pod/metadata-grpc-deployment-7d75ddb958-gtrt9          Error: ErrImagePull
Step #1 - "verify": 9m18s       Warning   Failed             pod/metadata-grpc-deployment-7d75ddb958-gtrt9          Error: ImagePullBackOff
Step #1 - "verify": 27m         Warning   Failed             pod/metadata-writer-7c85b864fb-2sglk                   Failed to pull image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/metadatawriter:0.5.1": rpc error: code = Unknown desc = Error response from daemon: manifest for gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/metadatawriter:0.5.1 not found
Step #1 - "verify": 27m         Warning   Failed             pod/metadata-writer-7c85b864fb-2sglk                   Error: ErrImagePull
Step #1 - "verify": 9m8s        Warning   Failed             pod/metadata-writer-7c85b864fb-2sglk                   Error: ImagePullBackOff
Step #1 - "verify": 29m         Warning   FailedScheduling   pod/minio-76566f85c6-jdgcz                             pod has unbound immediate PersistentVolumeClaims (repeated 3 times)
Step #1 - "verify": 27m         Warning   Failed             pod/minio-76566f85c6-jdgcz                             Failed to pull image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/minio:0.5.1": rpc error: code = Unknown desc = Error response from daemon: manifest for gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/minio:0.5.1 not found
Step #1 - "verify": 27m         Warning   Failed             pod/minio-76566f85c6-jdgcz                             Error: ErrImagePull
Step #1 - "verify": 3m54s       Warning   Failed             pod/minio-76566f85c6-jdgcz                             Error: ImagePullBackOff
Step #1 - "verify": 27m         Warning   Failed             pod/ml-pipeline-59d49c8f68-qrk27                       Failed to pull image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/apiserver:0.5.1": rpc error: code = Unknown desc = Error response from daemon: manifest for gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/apiserver:0.5.1 not found
Step #1 - "verify": 27m         Warning   Failed             pod/ml-pipeline-59d49c8f68-qrk27                       Error: ErrImagePull
Step #1 - "verify": 4m11s       Warning   Failed             pod/ml-pipeline-59d49c8f68-qrk27                       Error: ImagePullBackOff
Step #1 - "verify": 27m         Warning   Failed             pod/ml-pipeline-persistenceagent-58b4c68c67-sdghx      Failed to pull image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/persistenceagent:0.5.1": rpc error: code = Unknown desc = Error response from daemon: manifest for gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/persistenceagent:0.5.1 not found
Step #1 - "verify": 27m         Warning   Failed             pod/ml-pipeline-persistenceagent-58b4c68c67-sdghx      Error: ErrImagePull
Step #1 - "verify": 4m7s        Warning   Failed             pod/ml-pipeline-persistenceagent-58b4c68c67-sdghx      Error: ImagePullBackOff
Step #1 - "verify": 27m         Warning   Failed             pod/ml-pipeline-scheduledworkflow-ddc5f6b8c-8snzn      Failed to pull image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/scheduledworkflow:0.5.1": rpc error: code = Unknown desc = Error response from daemon: manifest for gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/scheduledworkflow:0.5.1 not found
Step #1 - "verify": 27m         Warning   Failed             pod/ml-pipeline-scheduledworkflow-ddc5f6b8c-8snzn      Error: ErrImagePull
Step #1 - "verify": 4m15s       Warning   Failed             pod/ml-pipeline-scheduledworkflow-ddc5f6b8c-8snzn      Error: ImagePullBackOff
Step #1 - "verify": 27m         Warning   Failed             pod/ml-pipeline-ui-6cb987c9df-ljqwj                    Failed to pull image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/frontend:0.5.1": rpc error: code = Unknown desc = Error response from daemon: manifest for gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/frontend:0.5.1 not found
Step #1 - "verify": 27m         Warning   Failed             pod/ml-pipeline-ui-6cb987c9df-ljqwj                    Error: ErrImagePull
Step #1 - "verify": 4m7s        Warning   Failed             pod/ml-pipeline-ui-6cb987c9df-ljqwj                    Error: ImagePullBackOff
Step #1 - "verify": 27m         Warning   Failed             pod/ml-pipeline-viewer-crd-58bbff9745-wdthk            Failed to pull image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/viewercrd:0.5.1": rpc error: code = Unknown desc = Error response from daemon: manifest for gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/viewercrd:0.5.1 not found
Step #1 - "verify": 27m         Warning   Failed             pod/ml-pipeline-viewer-crd-58bbff9745-wdthk            Error: ErrImagePull
Step #1 - "verify": 4m10s       ERROR: (gcloud.builds.submit) build 9e47be82-510b-493c-bb23-de6d28c71089 completed with status "FAILURE"
Warning   Failed             pod/ml-pipeline-viewer-crd-58bbff9745-wdthk            Error: ImagePullBackOff
Step #1 - "verify": 27m         Warning   Failed             pod/ml-pipeline-visualizationserver-574774d768-rr8gb   Failed to pull image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/visualizationserver:0.5.1": rpc error: code = Unknown desc = Error response from daemon: manifest for gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/visualizationserver:0.5.1 not found
Step #1 - "verify": 27m         Warning   Failed             pod/ml-pipeline-visualizationserver-574774d768-rr8gb   Error: ErrImagePull
Step #1 - "verify": 9m13s       Warning   Failed             pod/ml-pipeline-visualizationserver-574774d768-rr8gb   Error: ImagePullBackOff
Step #1 - "verify": 29m         Warning   FailedScheduling   pod/mysql-9cb48b5cc-b8h54                              pod has unbound immediate PersistentVolumeClaims (repeated 3 times)
Step #1 - "verify": 27m         Warning   Failed             pod/mysql-9cb48b5cc-b8h54                              Failed to pull image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/mysql:0.5.1": rpc error: code = Unknown desc = Error response from daemon: manifest for gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/mysql:0.5.1 not found
Step #1 - "verify": 27m         Warning   Failed             pod/mysql-9cb48b5cc-b8h54                              Error: ErrImagePull
Step #1 - "verify": 8m50s       Warning   Failed             pod/mysql-9cb48b5cc-b8h54                              Error: ImagePullBackOff
Step #1 - "verify": 27m         Warning   Failed             pod/proxy-agent-75d9f4659d-l6rb9                       Failed to pull image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/proxyagent:0.5.1": rpc error: code = Unknown desc = Error response from daemon: manifest for gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/proxyagent:0.5.1 not found
Step #1 - "verify": 27m         Warning   Failed             pod/proxy-agent-75d9f4659d-l6rb9                       Error: ErrImagePull
Step #1 - "verify": 4m13s       Warning   Failed             pod/proxy-agent-75d9f4659d-l6rb9                       Error: ImagePullBackOff
Step #1 - "verify": 28m         Warning   Failed             pod/workflow-controller-5855876fb9-78h78               Failed to pull image "gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/argoworkflowcontroller:0.5.1": rpc error: code = Unknown desc = Error response from daemon: manifest for gcr.io/ml-pipeline-test/hosted/1967d25c27f6201849dc3941fe5d393ac5f6e09c/argoworkflowcontroller:0.5.1 not found
Step #1 - "verify": 28m         Warning   Failed             pod/workflow-controller-5855876fb9-78h78               Error: ErrImagePull
Step #1 - "verify": 4m13s       Warning   Failed             pod/workflow-controller-5855876fb9-78h78               Error: ImagePullBackOff
Step #1 - "verify": =====================
Step #1 - "verify": 
Step #1 - "verify": === VERIFICATION STATUS ===
Step #1 - "verify": FAILED
Step #1 - "verify": ===========================
Finished Step #1 - "verify"
ERROR
ERROR: build step 1 "gcr.io/cloud-marketplace-tools/k8s/dev:latest" failed: step exited with non-zero status: 1
--------------------------------------------------------------------------------

+ clean_up
+ set +e
+ set +x
Status of pods before clean up:
... skipping 19 lines ...