This job view page is being replaced by Spyglass soon. Check out the new job view.
ResultFAILURE
Tests 0 failed / 0 succeeded
Started2020-07-10 11:10
Elapsed12m48s
Revision
Buildere38bd037-c29d-11ea-8304-86edacb83982
Refs master:b47163db
23150:d90de451
infra-commit3b296042d
repogithub.com/helm/charts
repo-commitc8b28c0ab41bf3a7a01b3a41cb9fba20d59dbb1f
repos{u'github.com/helm/charts': u'master:b47163db0c72f21f52c26e90e28395750f537853,23150:d90de451127cb4ae8088200161a26340148834df'}

No Test Failures!


Error lines from build-log.txt

... skipping 362 lines ...
I0710 11:11:43.540] Deleting namespace 'metheus-operator-presubmit-23150-1281546306048757760-etdg1eaq1x'...
I0710 11:11:43.818] namespace "metheus-operator-presubmit-23150-1281546306048757760-etdg1eaq1x" deleted
I0710 11:12:11.981] Namespace 'metheus-operator-presubmit-23150-1281546306048757760-etdg1eaq1x' terminated.
I0710 11:12:11.981] 
I0710 11:12:11.981] Installing chart with values file 'stable/prometheus-operator/ci/02-test-without-crds-values.yaml'...
I0710 11:12:11.981] 
I0710 11:22:51.459] Error: release prometheus-operator-hu8ukzt8ir failed: timed out waiting for the condition
I0710 11:22:51.460] ========================================================================================================================
I0710 11:22:51.460] ........................................................................................................................
I0710 11:22:51.460] ==> Events of namespace metheus-operator-presubmit-23150-1281546306048757760-hu8ukzt8ir
I0710 11:22:51.460] ........................................................................................................................
I0710 11:22:52.329] LAST SEEN   TYPE      REASON              OBJECT                                                                    SUBOBJECT                                     SOURCE                                                     MESSAGE                                                                                                                                                                                                    FIRST SEEN   COUNT   NAME
I0710 11:22:52.330] 10m         Normal    Scheduled           pod/alertmanager-prometheus-operator-hu8ukz-alertmanager-0                                                              default-scheduler                                          Successfully assigned metheus-operator-presubmit-23150-1281546306048757760-hu8ukzt8ir/alertmanager-prometheus-operator-hu8ukz-alertmanager-0 to gke-charts-testing-1-default-pool-7352761b-ts2k            10m          1       alertmanager-prometheus-operator-hu8ukz-alertmanager-0.16205fc5f6ee2026
... skipping 176 lines ...
I0710 11:22:53.092] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
I0710 11:22:53.549] ------------------------------------------------------------------------------------------------------------------------
I0710 11:22:53.549] ==> Logs of container alertmanager-prometheus-operator-hu8ukz-alertmanager-0
I0710 11:22:53.549] ------------------------------------------------------------------------------------------------------------------------
I0710 11:22:53.868] level=info ts=2020-07-10T11:12:56.088Z caller=main.go:216 msg="Starting Alertmanager" version="(version=0.21.0, branch=HEAD, revision=4c6c03ebfe21009c546e4d1e9b92c371d67c021d)"
I0710 11:22:53.868] level=info ts=2020-07-10T11:12:56.088Z caller=main.go:217 build_context="(go=go1.14.4, user=root@dee35927357f, date=20200617-08:54:02)"
I0710 11:22:53.869] level=warn ts=2020-07-10T11:12:56.172Z caller=cluster.go:228 component=cluster msg="failed to join cluster" err="1 error occurred:\n\t* Failed to resolve alertmanager-prometheus-operator-hu8ukz-alertmanager-0.alertmanager-operated.metheus-operator-presubmit-23150-1281546306048757760-hu8ukzt8ir.svc:9094: lookup alertmanager-prometheus-operator-hu8ukz-alertmanager-0.alertmanager-operated.metheus-operator-presubmit-23150-1281546306048757760-hu8ukzt8ir.svc on 10.39.240.10:53: no such host\n\n"
I0710 11:22:53.869] level=info ts=2020-07-10T11:12:56.172Z caller=cluster.go:230 component=cluster msg="will retry joining cluster every 10s"
I0710 11:22:53.869] level=warn ts=2020-07-10T11:12:56.172Z caller=main.go:307 msg="unable to join gossip mesh" err="1 error occurred:\n\t* Failed to resolve alertmanager-prometheus-operator-hu8ukz-alertmanager-0.alertmanager-operated.metheus-operator-presubmit-23150-1281546306048757760-hu8ukzt8ir.svc:9094: lookup alertmanager-prometheus-operator-hu8ukz-alertmanager-0.alertmanager-operated.metheus-operator-presubmit-23150-1281546306048757760-hu8ukzt8ir.svc on 10.39.240.10:53: no such host\n\n"
I0710 11:22:53.870] level=info ts=2020-07-10T11:12:56.172Z caller=cluster.go:623 component=cluster msg="Waiting for gossip to settle..." interval=2s
I0710 11:22:53.870] level=info ts=2020-07-10T11:12:56.200Z caller=coordinator.go:119 component=configuration msg="Loading configuration file" file=/etc/alertmanager/config/alertmanager.yaml
I0710 11:22:53.870] level=info ts=2020-07-10T11:12:56.201Z caller=coordinator.go:131 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/config/alertmanager.yaml
I0710 11:22:53.870] level=info ts=2020-07-10T11:12:56.204Z caller=main.go:485 msg=Listening address=:9093
I0710 11:22:53.870] level=info ts=2020-07-10T11:12:58.172Z caller=cluster.go:648 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000162286s
I0710 11:22:53.871] level=info ts=2020-07-10T11:13:06.173Z caller=cluster.go:640 component=cluster msg="gossip settled; proceeding" elapsed=10.001023282s
... skipping 131 lines ...
I0710 11:22:55.259] level=warn ts=2020-07-10T11:13:01.881791036Z caller=operator.go:1199 component=prometheusoperator msg="prometheus key=metheus-operator-presubmit-23150-1281546306048757760-hu8ukzt8ir/prometheus-operator-hu8ukz-prometheus, field spec.baseImage is deprecated, 'spec.image' field should be used instead"
I0710 11:22:55.259] level=info ts=2020-07-10T11:13:01.88186442Z caller=operator.go:1094 component=prometheusoperator msg="sync prometheus" key=metheus-operator-presubmit-23150-1281546306048757760-hu8ukzt8ir/prometheus-operator-hu8ukz-prometheus
I0710 11:22:55.260] level=info ts=2020-07-10T11:13:02.203828345Z caller=operator.go:1094 component=prometheusoperator msg="sync prometheus" key=metheus-operator-presubmit-23150-1281546306048757760-hu8ukzt8ir/prometheus-operator-hu8ukz-prometheus
I0710 11:22:55.260] level=info ts=2020-07-10T11:13:02.31386898Z caller=operator.go:1094 component=prometheusoperator msg="sync prometheus" key=metheus-operator-presubmit-23150-1281546306048757760-hu8ukzt8ir/prometheus-operator-hu8ukz-prometheus
I0710 11:22:55.260] level=info ts=2020-07-10T11:13:02.441373963Z caller=operator.go:1094 component=prometheusoperator msg="sync prometheus" key=metheus-operator-presubmit-23150-1281546306048757760-hu8ukzt8ir/prometheus-operator-hu8ukz-prometheus
I0710 11:22:55.261] level=info ts=2020-07-10T11:13:14.42807759Z caller=operator.go:1094 component=prometheusoperator msg="sync prometheus" key=metheus-operator-presubmit-23150-1281546306048757760-hu8ukzt8ir/prometheus-operator-hu8ukz-prometheus
I0710 11:22:55.261] E0710 11:18:47.122432       1 reflector.go:280] github.com/coreos/prometheus-operator/pkg/prometheus/operator.go:479: Failed to watch *v1.ConfigMap: expected resource version to have 2 parts to match the number of ListerWatchers
I0710 11:22:55.261] E0710 11:19:55.141826       1 reflector.go:280] github.com/coreos/prometheus-operator/pkg/prometheus/operator.go:481: Failed to watch *v1.StatefulSet: expected resource version to have 2 parts to match the number of ListerWatchers
I0710 11:22:55.261] E0710 11:20:05.650713       1 reflector.go:280] github.com/coreos/prometheus-operator/pkg/thanos/operator.go:318: Failed to watch *v1.ConfigMap: expected resource version to have 2 parts to match the number of ListerWatchers
I0710 11:22:55.262] E0710 11:20:35.124058       1 reflector.go:280] github.com/coreos/prometheus-operator/pkg/prometheus/operator.go:480: Failed to watch *v1.Secret: expected resource version to have 2 parts to match the number of ListerWatchers
I0710 11:22:55.262] E0710 11:21:13.652104       1 reflector.go:280] github.com/coreos/prometheus-operator/pkg/thanos/operator.go:321: Failed to watch *v1.StatefulSet: expected resource version to have 2 parts to match the number of ListerWatchers
I0710 11:22:55.263] E0710 11:22:33.635210       1 reflector.go:280] github.com/coreos/prometheus-operator/pkg/alertmanager/operator.go:238: Failed to watch *v1.StatefulSet: expected resource version to have 2 parts to match the number of ListerWatchers
I0710 11:22:55.263] level=warn ts=2020-07-10T11:22:51.626595326Z caller=operator.go:516 component=alertmanageroperator msg="alertmanager key=metheus-operator-presubmit-23150-1281546306048757760-hu8ukzt8ir/prometheus-operator-hu8ukz-alertmanager, field spec.baseImage is deprecated, 'spec.image' field should be used instead"
I0710 11:22:55.263] level=info ts=2020-07-10T11:22:51.626687514Z caller=operator.go:459 component=alertmanageroperator msg="sync alertmanager" key=metheus-operator-presubmit-23150-1281546306048757760-hu8ukzt8ir/prometheus-operator-hu8ukz-alertmanager
I0710 11:22:55.264] ------------------------------------------------------------------------------------------------------------------------
I0710 11:22:55.264] <== Logs of container prometheus-operator-hu8ukz-operator-564fc86cf-nq4cj
I0710 11:22:55.264] ------------------------------------------------------------------------------------------------------------------------
I0710 11:22:55.264] ------------------------------------------------------------------------------------------------------------------------
... skipping 933 lines ...
I0710 11:23:03.647]       --storage.tsdb.no-lockfile
I0710 11:23:03.647]       --web.external-url=http://prometheus-operator-hu8ukz-prometheus.metheus-operator-presubmit-23150-1281546306048757760-hu8ukzt8ir:9090
I0710 11:23:03.647]       --web.route-prefix=/
I0710 11:23:03.647]     State:       Running
I0710 11:23:03.648]       Started:   Fri, 10 Jul 2020 11:13:11 +0000
I0710 11:23:03.648]     Last State:  Terminated
I0710 11:23:03.648]       Reason:    Error
I0710 11:23:03.648]       Message:   Start listening for connections" address=0.0.0.0:9090
I0710 11:23:03.648] level=info ts=2020-07-10T11:13:10.081Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile"
I0710 11:23:03.648] level=info ts=2020-07-10T11:13:10.082Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0
I0710 11:23:03.649] level=info ts=2020-07-10T11:13:10.082Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=431.521µs
I0710 11:23:03.649] level=info ts=2020-07-10T11:13:10.084Z caller=main.go:694 fs_type=EXT4_SUPER_MAGIC
I0710 11:23:03.649] level=info ts=2020-07-10T11:13:10.084Z caller=main.go:695 msg="TSDB started"
... skipping 5 lines ...
I0710 11:23:03.650] level=info ts=2020-07-10T11:13:10.084Z caller=main.go:543 msg="Scrape discovery manager stopped"
I0710 11:23:03.650] level=info ts=2020-07-10T11:13:10.084Z caller=main.go:577 msg="Scrape manager stopped"
I0710 11:23:03.650] level=info ts=2020-07-10T11:13:10.084Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..."
I0710 11:23:03.651] level=info ts=2020-07-10T11:13:10.084Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped"
I0710 11:23:03.651] level=info ts=2020-07-10T11:13:10.087Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..."
I0710 11:23:03.651] level=info ts=2020-07-10T11:13:10.087Z caller=main.go:749 msg="Notifier manager stopped"
I0710 11:23:03.651] level=error ts=2020-07-10T11:13:10.088Z caller=main.go:758 err="error loading config from \"/etc/prometheus/config_out/prometheus.env.yaml\": couldn't load configuration (--config.file=\"/etc/prometheus/config_out/prometheus.env.yaml\"): open /etc/prometheus/config_out/prometheus.env.yaml: no such file or directory"
I0710 11:23:03.652] 
I0710 11:23:03.652]       Exit Code:    1
I0710 11:23:03.652]       Started:      Fri, 10 Jul 2020 11:13:09 +0000
I0710 11:23:03.652]       Finished:     Fri, 10 Jul 2020 11:13:10 +0000
I0710 11:23:03.652]     Ready:          True
I0710 11:23:03.652]     Restart Count:  1
... skipping 137 lines ...
I0710 11:23:04.592] level=info ts=2020-07-10T11:13:15.199Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml
I0710 11:23:04.593] level=info ts=2020-07-10T11:13:15.204Z caller=kubernetes.go:253 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"
I0710 11:23:04.593] level=info ts=2020-07-10T11:13:15.205Z caller=kubernetes.go:253 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"
I0710 11:23:04.593] level=info ts=2020-07-10T11:13:15.206Z caller=kubernetes.go:253 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"
I0710 11:23:04.593] level=info ts=2020-07-10T11:13:15.207Z caller=kubernetes.go:253 component="discovery manager notify" discovery=k8s msg="Using pod service account via in-cluster config"
I0710 11:23:04.593] level=info ts=2020-07-10T11:13:15.278Z caller=main.go:827 msg="Completed loading of configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml
I0710 11:23:04.594] level=warn ts=2020-07-10T11:13:17.892Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.99\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.595] level=warn ts=2020-07-10T11:13:17.893Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.9, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.9\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.596] level=warn ts=2020-07-10T11:13:17.894Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.5, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.5\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.598] level=warn ts=2020-07-10T11:13:22.281Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletTooManyPods\nexpr: max by(node) (max by(instance) (kubelet_running_pod_count{job=\"kubelet\",metrics_path=\"/metrics\"})\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\n  / max by(node) (kube_node_status_capacity_pods{job=\"kube-state-metrics\"} != 1) >\n  0.95\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet '{{ $labels.node }}' is running at {{ $value | humanizePercentage\n    }} of its Pod capacity.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubelettoomanypods\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.599] level=warn ts=2020-07-10T11:13:22.286Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletPodStartUpLatencyHigh\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\"kubelet\",metrics_path=\"/metrics\"}[5m])))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"}\n  > 60\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet Pod startup 99th percentile latency is {{ $value }} seconds on\n    node {{ $labels.node }}.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.600] level=warn ts=2020-07-10T11:13:47.908Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.99\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.601] level=warn ts=2020-07-10T11:13:47.924Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.9, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.9\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.602] level=warn ts=2020-07-10T11:13:47.940Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.5, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.5\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.603] level=warn ts=2020-07-10T11:13:52.284Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletTooManyPods\nexpr: max by(node) (max by(instance) (kubelet_running_pod_count{job=\"kubelet\",metrics_path=\"/metrics\"})\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\n  / max by(node) (kube_node_status_capacity_pods{job=\"kube-state-metrics\"} != 1) >\n  0.95\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet '{{ $labels.node }}' is running at {{ $value | humanizePercentage\n    }} of its Pod capacity.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubelettoomanypods\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.604] level=warn ts=2020-07-10T11:13:52.315Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletPodStartUpLatencyHigh\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\"kubelet\",metrics_path=\"/metrics\"}[5m])))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"}\n  > 60\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet Pod startup 99th percentile latency is {{ $value }} seconds on\n    node {{ $labels.node }}.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.605] level=warn ts=2020-07-10T11:14:17.926Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.99\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.606] level=warn ts=2020-07-10T11:14:17.961Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.9, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.9\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.607] level=warn ts=2020-07-10T11:14:17.987Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.5, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.5\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.626] level=warn ts=2020-07-10T11:14:22.688Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletTooManyPods\nexpr: max by(node) (max by(instance) (kubelet_running_pod_count{job=\"kubelet\",metrics_path=\"/metrics\"})\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\n  / max by(node) (kube_node_status_capacity_pods{job=\"kube-state-metrics\"} != 1) >\n  0.95\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet '{{ $labels.node }}' is running at {{ $value | humanizePercentage\n    }} of its Pod capacity.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubelettoomanypods\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.627] level=warn ts=2020-07-10T11:14:22.751Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletPodStartUpLatencyHigh\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\"kubelet\",metrics_path=\"/metrics\"}[5m])))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"}\n  > 60\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet Pod startup 99th percentile latency is {{ $value }} seconds on\n    node {{ $labels.node }}.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.628] level=warn ts=2020-07-10T11:14:47.915Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.99\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.629] level=warn ts=2020-07-10T11:14:47.936Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.9, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.9\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.630] level=warn ts=2020-07-10T11:14:47.953Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.5, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.5\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.632] level=warn ts=2020-07-10T11:14:52.284Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletTooManyPods\nexpr: max by(node) (max by(instance) (kubelet_running_pod_count{job=\"kubelet\",metrics_path=\"/metrics\"})\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\n  / max by(node) (kube_node_status_capacity_pods{job=\"kube-state-metrics\"} != 1) >\n  0.95\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet '{{ $labels.node }}' is running at {{ $value | humanizePercentage\n    }} of its Pod capacity.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubelettoomanypods\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.633] level=warn ts=2020-07-10T11:14:52.321Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletPodStartUpLatencyHigh\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\"kubelet\",metrics_path=\"/metrics\"}[5m])))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"}\n  > 60\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet Pod startup 99th percentile latency is {{ $value }} seconds on\n    node {{ $labels.node }}.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.634] level=warn ts=2020-07-10T11:15:17.928Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.99\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.635] level=warn ts=2020-07-10T11:15:17.947Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.9, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.9\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.636] level=warn ts=2020-07-10T11:15:17.963Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.5, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.5\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.637] level=warn ts=2020-07-10T11:15:22.283Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletTooManyPods\nexpr: max by(node) (max by(instance) (kubelet_running_pod_count{job=\"kubelet\",metrics_path=\"/metrics\"})\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\n  / max by(node) (kube_node_status_capacity_pods{job=\"kube-state-metrics\"} != 1) >\n  0.95\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet '{{ $labels.node }}' is running at {{ $value | humanizePercentage\n    }} of its Pod capacity.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubelettoomanypods\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.638] level=warn ts=2020-07-10T11:15:22.322Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletPodStartUpLatencyHigh\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\"kubelet\",metrics_path=\"/metrics\"}[5m])))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"}\n  > 60\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet Pod startup 99th percentile latency is {{ $value }} seconds on\n    node {{ $labels.node }}.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.639] level=warn ts=2020-07-10T11:15:47.914Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.99\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.640] level=warn ts=2020-07-10T11:15:47.938Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.9, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.9\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.641] level=warn ts=2020-07-10T11:15:47.962Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.5, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.5\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.643] level=warn ts=2020-07-10T11:15:52.284Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletTooManyPods\nexpr: max by(node) (max by(instance) (kubelet_running_pod_count{job=\"kubelet\",metrics_path=\"/metrics\"})\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\n  / max by(node) (kube_node_status_capacity_pods{job=\"kube-state-metrics\"} != 1) >\n  0.95\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet '{{ $labels.node }}' is running at {{ $value | humanizePercentage\n    }} of its Pod capacity.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubelettoomanypods\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.644] level=warn ts=2020-07-10T11:15:52.332Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletPodStartUpLatencyHigh\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\"kubelet\",metrics_path=\"/metrics\"}[5m])))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"}\n  > 60\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet Pod startup 99th percentile latency is {{ $value }} seconds on\n    node {{ $labels.node }}.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.645] level=warn ts=2020-07-10T11:16:17.917Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.99\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.646] level=warn ts=2020-07-10T11:16:17.940Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.9, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.9\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.647] level=warn ts=2020-07-10T11:16:17.962Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.5, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.5\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.648] level=warn ts=2020-07-10T11:16:22.284Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletTooManyPods\nexpr: max by(node) (max by(instance) (kubelet_running_pod_count{job=\"kubelet\",metrics_path=\"/metrics\"})\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\n  / max by(node) (kube_node_status_capacity_pods{job=\"kube-state-metrics\"} != 1) >\n  0.95\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet '{{ $labels.node }}' is running at {{ $value | humanizePercentage\n    }} of its Pod capacity.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubelettoomanypods\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.649] level=warn ts=2020-07-10T11:16:22.330Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletPodStartUpLatencyHigh\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\"kubelet\",metrics_path=\"/metrics\"}[5m])))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"}\n  > 60\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet Pod startup 99th percentile latency is {{ $value }} seconds on\n    node {{ $labels.node }}.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.650] level=warn ts=2020-07-10T11:16:47.915Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.99\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.651] level=warn ts=2020-07-10T11:16:47.934Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.9, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.9\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.652] level=warn ts=2020-07-10T11:16:47.949Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.5, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.5\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.653] level=warn ts=2020-07-10T11:16:52.284Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletTooManyPods\nexpr: max by(node) (max by(instance) (kubelet_running_pod_count{job=\"kubelet\",metrics_path=\"/metrics\"})\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\n  / max by(node) (kube_node_status_capacity_pods{job=\"kube-state-metrics\"} != 1) >\n  0.95\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet '{{ $labels.node }}' is running at {{ $value | humanizePercentage\n    }} of its Pod capacity.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubelettoomanypods\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.655] level=warn ts=2020-07-10T11:16:52.329Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletPodStartUpLatencyHigh\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\"kubelet\",metrics_path=\"/metrics\"}[5m])))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"}\n  > 60\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet Pod startup 99th percentile latency is {{ $value }} seconds on\n    node {{ $labels.node }}.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.656] level=warn ts=2020-07-10T11:17:17.915Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.99\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.657] level=warn ts=2020-07-10T11:17:17.934Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.9, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.9\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.658] level=warn ts=2020-07-10T11:17:17.952Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.5, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.5\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.659] level=warn ts=2020-07-10T11:17:22.284Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletTooManyPods\nexpr: max by(node) (max by(instance) (kubelet_running_pod_count{job=\"kubelet\",metrics_path=\"/metrics\"})\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\n  / max by(node) (kube_node_status_capacity_pods{job=\"kube-state-metrics\"} != 1) >\n  0.95\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet '{{ $labels.node }}' is running at {{ $value | humanizePercentage\n    }} of its Pod capacity.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubelettoomanypods\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.661] level=warn ts=2020-07-10T11:17:22.329Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletPodStartUpLatencyHigh\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\"kubelet\",metrics_path=\"/metrics\"}[5m])))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"}\n  > 60\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet Pod startup 99th percentile latency is {{ $value }} seconds on\n    node {{ $labels.node }}.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.662] level=warn ts=2020-07-10T11:17:47.916Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.99\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.663] level=warn ts=2020-07-10T11:17:47.939Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.9, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.9\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.664] level=warn ts=2020-07-10T11:17:47.955Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.5, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.5\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.665] level=warn ts=2020-07-10T11:17:52.284Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletTooManyPods\nexpr: max by(node) (max by(instance) (kubelet_running_pod_count{job=\"kubelet\",metrics_path=\"/metrics\"})\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\n  / max by(node) (kube_node_status_capacity_pods{job=\"kube-state-metrics\"} != 1) >\n  0.95\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet '{{ $labels.node }}' is running at {{ $value | humanizePercentage\n    }} of its Pod capacity.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubelettoomanypods\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.667] level=warn ts=2020-07-10T11:17:52.325Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletPodStartUpLatencyHigh\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\"kubelet\",metrics_path=\"/metrics\"}[5m])))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"}\n  > 60\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet Pod startup 99th percentile latency is {{ $value }} seconds on\n    node {{ $labels.node }}.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.668] level=warn ts=2020-07-10T11:18:17.917Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.99\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.669] level=warn ts=2020-07-10T11:18:17.941Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.9, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.9\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.670] level=warn ts=2020-07-10T11:18:17.957Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.5, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.5\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.672] level=warn ts=2020-07-10T11:18:22.283Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletTooManyPods\nexpr: max by(node) (max by(instance) (kubelet_running_pod_count{job=\"kubelet\",metrics_path=\"/metrics\"})\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\n  / max by(node) (kube_node_status_capacity_pods{job=\"kube-state-metrics\"} != 1) >\n  0.95\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet '{{ $labels.node }}' is running at {{ $value | humanizePercentage\n    }} of its Pod capacity.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubelettoomanypods\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.673] level=warn ts=2020-07-10T11:18:22.335Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletPodStartUpLatencyHigh\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\"kubelet\",metrics_path=\"/metrics\"}[5m])))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"}\n  > 60\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet Pod startup 99th percentile latency is {{ $value }} seconds on\n    node {{ $labels.node }}.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.674] level=warn ts=2020-07-10T11:18:47.923Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.99\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.675] level=warn ts=2020-07-10T11:18:47.953Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.9, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.9\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.676] level=warn ts=2020-07-10T11:18:47.976Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.5, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.5\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.677] level=warn ts=2020-07-10T11:18:52.284Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletTooManyPods\nexpr: max by(node) (max by(instance) (kubelet_running_pod_count{job=\"kubelet\",metrics_path=\"/metrics\"})\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\n  / max by(node) (kube_node_status_capacity_pods{job=\"kube-state-metrics\"} != 1) >\n  0.95\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet '{{ $labels.node }}' is running at {{ $value | humanizePercentage\n    }} of its Pod capacity.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubelettoomanypods\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.679] level=warn ts=2020-07-10T11:18:52.336Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletPodStartUpLatencyHigh\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\"kubelet\",metrics_path=\"/metrics\"}[5m])))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"}\n  > 60\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet Pod startup 99th percentile latency is {{ $value }} seconds on\n    node {{ $labels.node }}.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.680] level=warn ts=2020-07-10T11:19:17.922Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.99\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.681] level=warn ts=2020-07-10T11:19:17.950Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.9, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.9\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.682] level=warn ts=2020-07-10T11:19:17.970Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.5, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.5\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.683] level=warn ts=2020-07-10T11:19:22.284Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletTooManyPods\nexpr: max by(node) (max by(instance) (kubelet_running_pod_count{job=\"kubelet\",metrics_path=\"/metrics\"})\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\n  / max by(node) (kube_node_status_capacity_pods{job=\"kube-state-metrics\"} != 1) >\n  0.95\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet '{{ $labels.node }}' is running at {{ $value | humanizePercentage\n    }} of its Pod capacity.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubelettoomanypods\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.684] level=warn ts=2020-07-10T11:19:22.332Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletPodStartUpLatencyHigh\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\"kubelet\",metrics_path=\"/metrics\"}[5m])))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"}\n  > 60\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet Pod startup 99th percentile latency is {{ $value }} seconds on\n    node {{ $labels.node }}.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.685] level=warn ts=2020-07-10T11:19:47.924Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.99\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.686] level=warn ts=2020-07-10T11:19:47.945Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.9, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.9\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.687] level=warn ts=2020-07-10T11:19:47.964Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.5, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.5\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.688] level=warn ts=2020-07-10T11:19:52.284Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletTooManyPods\nexpr: max by(node) (max by(instance) (kubelet_running_pod_count{job=\"kubelet\",metrics_path=\"/metrics\"})\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\n  / max by(node) (kube_node_status_capacity_pods{job=\"kube-state-metrics\"} != 1) >\n  0.95\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet '{{ $labels.node }}' is running at {{ $value | humanizePercentage\n    }} of its Pod capacity.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubelettoomanypods\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.690] level=warn ts=2020-07-10T11:19:52.330Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletPodStartUpLatencyHigh\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\"kubelet\",metrics_path=\"/metrics\"}[5m])))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"}\n  > 60\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet Pod startup 99th percentile latency is {{ $value }} seconds on\n    node {{ $labels.node }}.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.691] level=warn ts=2020-07-10T11:20:17.929Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.99\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.692] level=warn ts=2020-07-10T11:20:17.955Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.9, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.9\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.693] level=warn ts=2020-07-10T11:20:17.977Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.5, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.5\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.694] level=warn ts=2020-07-10T11:20:22.284Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletTooManyPods\nexpr: max by(node) (max by(instance) (kubelet_running_pod_count{job=\"kubelet\",metrics_path=\"/metrics\"})\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\n  / max by(node) (kube_node_status_capacity_pods{job=\"kube-state-metrics\"} != 1) >\n  0.95\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet '{{ $labels.node }}' is running at {{ $value | humanizePercentage\n    }} of its Pod capacity.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubelettoomanypods\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.695] level=warn ts=2020-07-10T11:20:22.333Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletPodStartUpLatencyHigh\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\"kubelet\",metrics_path=\"/metrics\"}[5m])))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"}\n  > 60\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet Pod startup 99th percentile latency is {{ $value }} seconds on\n    node {{ $labels.node }}.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.697] level=warn ts=2020-07-10T11:20:47.922Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.99\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.698] level=warn ts=2020-07-10T11:20:47.942Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.9, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.9\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.699] level=warn ts=2020-07-10T11:20:47.962Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.5, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.5\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.700] level=warn ts=2020-07-10T11:20:52.284Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletTooManyPods\nexpr: max by(node) (max by(instance) (kubelet_running_pod_count{job=\"kubelet\",metrics_path=\"/metrics\"})\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\n  / max by(node) (kube_node_status_capacity_pods{job=\"kube-state-metrics\"} != 1) >\n  0.95\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet '{{ $labels.node }}' is running at {{ $value | humanizePercentage\n    }} of its Pod capacity.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubelettoomanypods\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.701] level=warn ts=2020-07-10T11:20:52.332Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletPodStartUpLatencyHigh\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\"kubelet\",metrics_path=\"/metrics\"}[5m])))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"}\n  > 60\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet Pod startup 99th percentile latency is {{ $value }} seconds on\n    node {{ $labels.node }}.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.702] level=warn ts=2020-07-10T11:21:17.917Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.99\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.703] level=warn ts=2020-07-10T11:21:17.939Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.9, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.9\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.704] level=warn ts=2020-07-10T11:21:17.959Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.5, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.5\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.706] level=warn ts=2020-07-10T11:21:22.284Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletTooManyPods\nexpr: max by(node) (max by(instance) (kubelet_running_pod_count{job=\"kubelet\",metrics_path=\"/metrics\"})\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\n  / max by(node) (kube_node_status_capacity_pods{job=\"kube-state-metrics\"} != 1) >\n  0.95\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet '{{ $labels.node }}' is running at {{ $value | humanizePercentage\n    }} of its Pod capacity.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubelettoomanypods\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.707] level=warn ts=2020-07-10T11:21:22.333Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletPodStartUpLatencyHigh\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\"kubelet\",metrics_path=\"/metrics\"}[5m])))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"}\n  > 60\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet Pod startup 99th percentile latency is {{ $value }} seconds on\n    node {{ $labels.node }}.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.708] level=warn ts=2020-07-10T11:21:47.929Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.99\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.709] level=warn ts=2020-07-10T11:21:47.964Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.9, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.9\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.710] level=warn ts=2020-07-10T11:21:47.993Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.5, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.5\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.712] level=warn ts=2020-07-10T11:21:52.285Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletTooManyPods\nexpr: max by(node) (max by(instance) (kubelet_running_pod_count{job=\"kubelet\",metrics_path=\"/metrics\"})\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\n  / max by(node) (kube_node_status_capacity_pods{job=\"kube-state-metrics\"} != 1) >\n  0.95\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet '{{ $labels.node }}' is running at {{ $value | humanizePercentage\n    }} of its Pod capacity.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubelettoomanypods\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.713] level=warn ts=2020-07-10T11:21:52.337Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletPodStartUpLatencyHigh\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\"kubelet\",metrics_path=\"/metrics\"}[5m])))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"}\n  > 60\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet Pod startup 99th percentile latency is {{ $value }} seconds on\n    node {{ $labels.node }}.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.714] level=warn ts=2020-07-10T11:22:17.920Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.99\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.715] level=warn ts=2020-07-10T11:22:17.942Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.9, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.9\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.716] level=warn ts=2020-07-10T11:22:17.970Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.5, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.5\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.718] level=warn ts=2020-07-10T11:22:22.284Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletTooManyPods\nexpr: max by(node) (max by(instance) (kubelet_running_pod_count{job=\"kubelet\",metrics_path=\"/metrics\"})\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\n  / max by(node) (kube_node_status_capacity_pods{job=\"kube-state-metrics\"} != 1) >\n  0.95\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet '{{ $labels.node }}' is running at {{ $value | humanizePercentage\n    }} of its Pod capacity.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubelettoomanypods\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.719] level=warn ts=2020-07-10T11:22:22.337Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletPodStartUpLatencyHigh\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\"kubelet\",metrics_path=\"/metrics\"}[5m])))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"}\n  > 60\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet Pod startup 99th percentile latency is {{ $value }} seconds on\n    node {{ $labels.node }}.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.720] level=warn ts=2020-07-10T11:22:47.921Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.99\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.721] level=warn ts=2020-07-10T11:22:47.946Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.9, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.9\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.722] level=warn ts=2020-07-10T11:22:47.969Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.5, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.5\"\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.724] level=warn ts=2020-07-10T11:22:52.284Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletTooManyPods\nexpr: max by(node) (max by(instance) (kubelet_running_pod_count{job=\"kubelet\",metrics_path=\"/metrics\"})\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\n  / max by(node) (kube_node_status_capacity_pods{job=\"kube-state-metrics\"} != 1) >\n  0.95\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet '{{ $labels.node }}' is running at {{ $value | humanizePercentage\n    }} of its Pod capacity.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubelettoomanypods\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.725] level=warn ts=2020-07-10T11:22:52.332Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletPodStartUpLatencyHigh\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\"kubelet\",metrics_path=\"/metrics\"}[5m])))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"}\n  > 60\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet Pod startup 99th percentile latency is {{ $value }} seconds on\n    node {{ $labels.node }}.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh\n" err="found duplicate series for the match group {instance=\"10.138.0.58:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-7ad3yr-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.58:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-ts2k\", service=\"prometheus-operator-573dew-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:23:04.725] ------------------------------------------------------------------------------------------------------------------------
I0710 11:23:04.726] <== Logs of container prometheus-prometheus-operator-hu8ukz-prometheus-0
I0710 11:23:04.726] ------------------------------------------------------------------------------------------------------------------------
I0710 11:23:04.726] ------------------------------------------------------------------------------------------------------------------------
I0710 11:23:04.726] ==> Logs of container prometheus-prometheus-operator-hu8ukz-prometheus-0
I0710 11:23:04.726] ------------------------------------------------------------------------------------------------------------------------
I0710 11:23:04.990] ts=2020-07-10T11:13:10.197814616Z caller=main.go:85 msg="Starting prometheus-config-reloader version '0.38.1'."
I0710 11:23:04.990] level=error ts=2020-07-10T11:13:10.19926854Z caller=runutil.go:98 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post http://127.0.0.1:9090/-/reload: dial tcp 127.0.0.1:9090: connect: connection refused"
I0710 11:23:04.990] level=info ts=2020-07-10T11:13:15.278806786Z caller=reloader.go:289 msg="Prometheus reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml rule_dirs=
I0710 11:23:04.991] level=info ts=2020-07-10T11:13:15.278946151Z caller=reloader.go:157 msg="started watching config file and non-recursively rule dirs for changes" cfg=/etc/prometheus/config/prometheus.yaml.gz out=/etc/prometheus/config_out/prometheus.env.yaml dirs=
I0710 11:23:04.994] ------------------------------------------------------------------------------------------------------------------------
I0710 11:23:04.995] <== Logs of container prometheus-prometheus-operator-hu8ukz-prometheus-0
I0710 11:23:04.995] ------------------------------------------------------------------------------------------------------------------------
I0710 11:23:04.995] ------------------------------------------------------------------------------------------------------------------------
... skipping 6 lines ...
I0710 11:23:05.337] ========================================================================================================================
I0710 11:23:05.337] Deleting release 'prometheus-operator-hu8ukzt8ir'...
I0710 11:23:11.065] release "prometheus-operator-hu8ukzt8ir" deleted
I0710 11:23:11.073] Deleting namespace 'metheus-operator-presubmit-23150-1281546306048757760-hu8ukzt8ir'...
I0710 11:23:11.337] namespace "metheus-operator-presubmit-23150-1281546306048757760-hu8ukzt8ir" deleted
I0710 11:23:18.401] Namespace 'metheus-operator-presubmit-23150-1281546306048757760-hu8ukzt8ir' terminated.
I0710 11:23:18.402] Error linting and installing charts: Error processing charts
I0710 11:23:18.402] ------------------------------------------------------------------------------------------------------------------------
I0710 11:23:18.402]  ✖︎ prometheus-operator => (version: "8.16.2", path: "stable/prometheus-operator") > Error waiting for process: exit status 1
I0710 11:23:18.402] ------------------------------------------------------------------------------------------------------------------------
I0710 11:23:19.056] ca2aa67aac1ea7d75f9fc728c786b283aa42d8d5d9873ebdc739701d81dec358
W0710 11:23:19.074] Traceback (most recent call last):
W0710 11:23:19.074]   File "/workspace/./test-infra/jenkins/../scenarios/execute.py", line 50, in <module>
W0710 11:23:19.074]     main(ARGS.env, ARGS.cmd + ARGS.args)
W0710 11:23:19.074]   File "/workspace/./test-infra/jenkins/../scenarios/execute.py", line 41, in main
W0710 11:23:19.075]     check(*cmd)
W0710 11:23:19.075]   File "/workspace/./test-infra/jenkins/../scenarios/execute.py", line 30, in check
W0710 11:23:19.075]     subprocess.check_call(cmd)
W0710 11:23:19.075]   File "/usr/lib/python2.7/subprocess.py", line 190, in check_call
W0710 11:23:19.075]     raise CalledProcessError(retcode, cmd)
W0710 11:23:19.076] subprocess.CalledProcessError: Command '('./test/e2e.sh',)' returned non-zero exit status 1
E0710 11:23:19.076] Command failed
I0710 11:23:19.076] process 645 exited with code 1 after 12.5m
E0710 11:23:19.076] FAIL: pull-charts-e2e
I0710 11:23:19.077] Call:  gcloud auth activate-service-account --key-file=/etc/service-account/service-account.json
W0710 11:23:19.762] Activated service account credentials for: [pr-kubekins@kubernetes-jenkins-pull.iam.gserviceaccount.com]
I0710 11:23:19.825] process 2036 exited with code 0 after 0.0m
I0710 11:23:19.825] Call:  gcloud config get-value account
I0710 11:23:20.334] process 2050 exited with code 0 after 0.0m
I0710 11:23:20.335] Will upload results to gs://kubernetes-jenkins/pr-logs using pr-kubekins@kubernetes-jenkins-pull.iam.gserviceaccount.com
... skipping 28 lines ...