This job view page is being replaced by Spyglass soon. Check out the new job view.
ResultFAILURE
Tests 0 failed / 0 succeeded
Started2020-07-10 10:51
Elapsed12m37s
Revision
Builder491615a7-c29b-11ea-aac9-bed756b503a7
Refs master:b47163db
23150:d90de451
infra-commit1ac27d740
repogithub.com/helm/charts
repo-commitc8b28c0ab41bf3a7a01b3a41cb9fba20d59dbb1f
repos{u'github.com/helm/charts': u'master:b47163db0c72f21f52c26e90e28395750f537853,23150:d90de451127cb4ae8088200161a26340148834df'}

No Test Failures!


Error lines from build-log.txt

... skipping 362 lines ...
I0710 10:52:41.916] Deleting namespace 'metheus-operator-presubmit-23150-1281541524571557888-898a6ati4r'...
I0710 10:52:42.178] namespace "metheus-operator-presubmit-23150-1281541524571557888-898a6ati4r" deleted
I0710 10:52:48.243] Namespace 'metheus-operator-presubmit-23150-1281541524571557888-898a6ati4r' terminated.
I0710 10:52:48.243] 
I0710 10:52:48.243] Installing chart with values file 'stable/prometheus-operator/ci/02-test-without-crds-values.yaml'...
I0710 10:52:48.243] 
I0710 11:03:27.722] Error: release prometheus-operator-iju343rqod failed: timed out waiting for the condition
I0710 11:03:27.722] ========================================================================================================================
I0710 11:03:27.722] ........................................................................................................................
I0710 11:03:27.723] ==> Events of namespace metheus-operator-presubmit-23150-1281541524571557888-iju343rqod
I0710 11:03:27.723] ........................................................................................................................
I0710 11:03:28.395] LAST SEEN   TYPE      REASON              OBJECT                                                                    SUBOBJECT                                     SOURCE                                                     MESSAGE                                                                                                                                                                                                    FIRST SEEN   COUNT   NAME
I0710 11:03:28.396] 9m59s       Normal    Scheduled           pod/alertmanager-prometheus-operator-iju343-alertmanager-0                                                              default-scheduler                                          Successfully assigned metheus-operator-presubmit-23150-1281541524571557888-iju343rqod/alertmanager-prometheus-operator-iju343-alertmanager-0 to gke-charts-testing-1-default-pool-7352761b-4zq1            9m59s        1       alertmanager-prometheus-operator-iju343-alertmanager-0.16205eb765dd59d4
... skipping 177 lines ...
I0710 11:03:29.133] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
I0710 11:03:29.622] ------------------------------------------------------------------------------------------------------------------------
I0710 11:03:29.623] ==> Logs of container alertmanager-prometheus-operator-iju343-alertmanager-0
I0710 11:03:29.623] ------------------------------------------------------------------------------------------------------------------------
I0710 11:03:29.951] level=info ts=2020-07-10T10:53:34.945Z caller=main.go:216 msg="Starting Alertmanager" version="(version=0.21.0, branch=HEAD, revision=4c6c03ebfe21009c546e4d1e9b92c371d67c021d)"
I0710 11:03:29.952] level=info ts=2020-07-10T10:53:34.945Z caller=main.go:217 build_context="(go=go1.14.4, user=root@dee35927357f, date=20200617-08:54:02)"
I0710 11:03:29.952] level=warn ts=2020-07-10T10:53:35.097Z caller=cluster.go:228 component=cluster msg="failed to join cluster" err="1 error occurred:\n\t* Failed to resolve alertmanager-prometheus-operator-iju343-alertmanager-0.alertmanager-operated.metheus-operator-presubmit-23150-1281541524571557888-iju343rqod.svc:9094: lookup alertmanager-prometheus-operator-iju343-alertmanager-0.alertmanager-operated.metheus-operator-presubmit-23150-1281541524571557888-iju343rqod.svc on 10.39.240.10:53: no such host\n\n"
I0710 11:03:29.953] level=info ts=2020-07-10T10:53:35.097Z caller=cluster.go:230 component=cluster msg="will retry joining cluster every 10s"
I0710 11:03:29.954] level=warn ts=2020-07-10T10:53:35.097Z caller=main.go:307 msg="unable to join gossip mesh" err="1 error occurred:\n\t* Failed to resolve alertmanager-prometheus-operator-iju343-alertmanager-0.alertmanager-operated.metheus-operator-presubmit-23150-1281541524571557888-iju343rqod.svc:9094: lookup alertmanager-prometheus-operator-iju343-alertmanager-0.alertmanager-operated.metheus-operator-presubmit-23150-1281541524571557888-iju343rqod.svc on 10.39.240.10:53: no such host\n\n"
I0710 11:03:29.954] level=info ts=2020-07-10T10:53:35.097Z caller=cluster.go:623 component=cluster msg="Waiting for gossip to settle..." interval=2s
I0710 11:03:29.954] level=info ts=2020-07-10T10:53:35.134Z caller=coordinator.go:119 component=configuration msg="Loading configuration file" file=/etc/alertmanager/config/alertmanager.yaml
I0710 11:03:29.955] level=info ts=2020-07-10T10:53:35.134Z caller=coordinator.go:131 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/config/alertmanager.yaml
I0710 11:03:29.955] level=info ts=2020-07-10T10:53:35.138Z caller=main.go:485 msg=Listening address=:9093
I0710 11:03:29.955] level=info ts=2020-07-10T10:53:37.097Z caller=cluster.go:648 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000189703s
I0710 11:03:29.955] level=info ts=2020-07-10T10:53:45.098Z caller=cluster.go:640 component=cluster msg="gossip settled; proceeding" elapsed=10.001094689s
... skipping 130 lines ...
I0710 11:03:31.650] level=warn ts=2020-07-10T10:53:39.668995117Z caller=operator.go:1199 component=prometheusoperator msg="prometheus key=metheus-operator-presubmit-23150-1281541524571557888-iju343rqod/prometheus-operator-iju343-prometheus, field spec.baseImage is deprecated, 'spec.image' field should be used instead"
I0710 11:03:31.650] level=info ts=2020-07-10T10:53:39.669114085Z caller=operator.go:1094 component=prometheusoperator msg="sync prometheus" key=metheus-operator-presubmit-23150-1281541524571557888-iju343rqod/prometheus-operator-iju343-prometheus
I0710 11:03:31.651] level=info ts=2020-07-10T10:53:39.869502325Z caller=operator.go:459 component=alertmanageroperator msg="sync alertmanager" key=metheus-operator-presubmit-23150-1281541524571557888-iju343rqod/prometheus-operator-iju343-alertmanager
I0710 11:03:31.651] level=info ts=2020-07-10T10:53:39.889294504Z caller=operator.go:1094 component=prometheusoperator msg="sync prometheus" key=metheus-operator-presubmit-23150-1281541524571557888-iju343rqod/prometheus-operator-iju343-prometheus
I0710 11:03:31.651] level=info ts=2020-07-10T10:53:40.044998731Z caller=operator.go:1094 component=prometheusoperator msg="sync prometheus" key=metheus-operator-presubmit-23150-1281541524571557888-iju343rqod/prometheus-operator-iju343-prometheus
I0710 11:03:31.651] level=info ts=2020-07-10T10:53:54.55251146Z caller=operator.go:1094 component=prometheusoperator msg="sync prometheus" key=metheus-operator-presubmit-23150-1281541524571557888-iju343rqod/prometheus-operator-iju343-prometheus
I0710 11:03:31.652] E0710 10:59:42.889770       1 reflector.go:280] github.com/coreos/prometheus-operator/pkg/prometheus/operator.go:481: Failed to watch *v1.StatefulSet: expected resource version to have 2 parts to match the number of ListerWatchers
I0710 11:03:31.652] E0710 11:00:32.900804       1 reflector.go:280] github.com/coreos/prometheus-operator/pkg/prometheus/operator.go:480: Failed to watch *v1.Secret: expected resource version to have 2 parts to match the number of ListerWatchers
I0710 11:03:31.652] E0710 11:00:43.726632       1 reflector.go:280] github.com/coreos/prometheus-operator/pkg/thanos/operator.go:318: Failed to watch *v1.ConfigMap: expected resource version to have 2 parts to match the number of ListerWatchers
I0710 11:03:31.652] E0710 11:01:51.721585       1 reflector.go:280] github.com/coreos/prometheus-operator/pkg/thanos/operator.go:321: Failed to watch *v1.StatefulSet: expected resource version to have 2 parts to match the number of ListerWatchers
I0710 11:03:31.653] E0710 11:02:42.893226       1 reflector.go:280] github.com/coreos/prometheus-operator/pkg/prometheus/operator.go:479: Failed to watch *v1.ConfigMap: expected resource version to have 2 parts to match the number of ListerWatchers
I0710 11:03:31.653] E0710 11:03:11.498205       1 reflector.go:280] github.com/coreos/prometheus-operator/pkg/alertmanager/operator.go:238: Failed to watch *v1.StatefulSet: expected resource version to have 2 parts to match the number of ListerWatchers
I0710 11:03:31.653] level=warn ts=2020-07-10T11:03:29.488031748Z caller=operator.go:516 component=alertmanageroperator msg="alertmanager key=metheus-operator-presubmit-23150-1281541524571557888-iju343rqod/prometheus-operator-iju343-alertmanager, field spec.baseImage is deprecated, 'spec.image' field should be used instead"
I0710 11:03:31.654] level=info ts=2020-07-10T11:03:29.488111749Z caller=operator.go:459 component=alertmanageroperator msg="sync alertmanager" key=metheus-operator-presubmit-23150-1281541524571557888-iju343rqod/prometheus-operator-iju343-alertmanager
I0710 11:03:31.654] ------------------------------------------------------------------------------------------------------------------------
I0710 11:03:31.654] <== Logs of container prometheus-operator-iju343-operator-7f9f7bcdd6-shf2l
I0710 11:03:31.654] ------------------------------------------------------------------------------------------------------------------------
I0710 11:03:31.654] ------------------------------------------------------------------------------------------------------------------------
... skipping 933 lines ...
I0710 11:03:39.765]       --storage.tsdb.no-lockfile
I0710 11:03:39.765]       --web.external-url=http://prometheus-operator-iju343-prometheus.metheus-operator-presubmit-23150-1281541524571557888-iju343rqod:9090
I0710 11:03:39.766]       --web.route-prefix=/
I0710 11:03:39.766]     State:       Running
I0710 11:03:39.766]       Started:   Fri, 10 Jul 2020 10:53:49 +0000
I0710 11:03:39.766]     Last State:  Terminated
I0710 11:03:39.766]       Reason:    Error
I0710 11:03:39.766]       Message:   53:47.923Z caller=main.go:678 msg="Starting TSDB ..."
I0710 11:03:39.766] level=info ts=2020-07-10T10:53:47.933Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile"
I0710 11:03:39.767] level=info ts=2020-07-10T10:53:47.933Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0
I0710 11:03:39.767] level=info ts=2020-07-10T10:53:47.933Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=414.008µs
I0710 11:03:39.767] level=info ts=2020-07-10T10:53:47.935Z caller=main.go:694 fs_type=EXT4_SUPER_MAGIC
I0710 11:03:39.767] level=info ts=2020-07-10T10:53:47.935Z caller=main.go:695 msg="TSDB started"
... skipping 5 lines ...
I0710 11:03:39.768] level=info ts=2020-07-10T10:53:47.936Z caller=main.go:543 msg="Scrape discovery manager stopped"
I0710 11:03:39.768] level=info ts=2020-07-10T10:53:47.936Z caller=main.go:577 msg="Scrape manager stopped"
I0710 11:03:39.769] level=info ts=2020-07-10T10:53:47.936Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..."
I0710 11:03:39.769] level=info ts=2020-07-10T10:53:47.936Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped"
I0710 11:03:39.769] level=info ts=2020-07-10T10:53:47.936Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..."
I0710 11:03:39.769] level=info ts=2020-07-10T10:53:47.936Z caller=main.go:749 msg="Notifier manager stopped"
I0710 11:03:39.770] level=error ts=2020-07-10T10:53:47.937Z caller=main.go:758 err="error loading config from \"/etc/prometheus/config_out/prometheus.env.yaml\": couldn't load configuration (--config.file=\"/etc/prometheus/config_out/prometheus.env.yaml\"): open /etc/prometheus/config_out/prometheus.env.yaml: no such file or directory"
I0710 11:03:39.770] 
I0710 11:03:39.770]       Exit Code:    1
I0710 11:03:39.770]       Started:      Fri, 10 Jul 2020 10:53:47 +0000
I0710 11:03:39.770]       Finished:     Fri, 10 Jul 2020 10:53:47 +0000
I0710 11:03:39.770]     Ready:          True
I0710 11:03:39.770]     Restart Count:  1
... skipping 137 lines ...
I0710 11:03:40.697] level=info ts=2020-07-10T10:53:53.117Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml
I0710 11:03:40.697] level=info ts=2020-07-10T10:53:53.121Z caller=kubernetes.go:253 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"
I0710 11:03:40.697] level=info ts=2020-07-10T10:53:53.123Z caller=kubernetes.go:253 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"
I0710 11:03:40.697] level=info ts=2020-07-10T10:53:53.124Z caller=kubernetes.go:253 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"
I0710 11:03:40.698] level=info ts=2020-07-10T10:53:53.125Z caller=kubernetes.go:253 component="discovery manager notify" discovery=k8s msg="Using pod service account via in-cluster config"
I0710 11:03:40.698] level=info ts=2020-07-10T10:53:53.195Z caller=main.go:827 msg="Completed loading of configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml
I0710 11:03:40.699] level=warn ts=2020-07-10T10:54:06.937Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletTooManyPods\nexpr: max by(node) (max by(instance) (kubelet_running_pod_count{job=\"kubelet\",metrics_path=\"/metrics\"})\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\n  / max by(node) (kube_node_status_capacity_pods{job=\"kube-state-metrics\"} != 1) >\n  0.95\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet '{{ $labels.node }}' is running at {{ $value | humanizePercentage\n    }} of its Pod capacity.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubelettoomanypods\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.701] level=warn ts=2020-07-10T10:54:06.951Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletPodStartUpLatencyHigh\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\"kubelet\",metrics_path=\"/metrics\"}[5m])))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"}\n  > 60\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet Pod startup 99th percentile latency is {{ $value }} seconds on\n    node {{ $labels.node }}.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.702] level=warn ts=2020-07-10T10:54:11.330Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.99\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.703] level=warn ts=2020-07-10T10:54:11.337Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.9, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.9\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.704] level=warn ts=2020-07-10T10:54:11.344Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.5, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.5\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.705] level=warn ts=2020-07-10T10:54:36.940Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletTooManyPods\nexpr: max by(node) (max by(instance) (kubelet_running_pod_count{job=\"kubelet\",metrics_path=\"/metrics\"})\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\n  / max by(node) (kube_node_status_capacity_pods{job=\"kube-state-metrics\"} != 1) >\n  0.95\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet '{{ $labels.node }}' is running at {{ $value | humanizePercentage\n    }} of its Pod capacity.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubelettoomanypods\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.706] level=warn ts=2020-07-10T10:54:36.975Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletPodStartUpLatencyHigh\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\"kubelet\",metrics_path=\"/metrics\"}[5m])))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"}\n  > 60\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet Pod startup 99th percentile latency is {{ $value }} seconds on\n    node {{ $labels.node }}.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.707] level=warn ts=2020-07-10T10:54:41.338Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.99\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.708] level=warn ts=2020-07-10T10:54:41.355Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.9, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.9\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.709] level=warn ts=2020-07-10T10:54:41.373Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.5, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.5\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.711] level=warn ts=2020-07-10T10:55:06.945Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletTooManyPods\nexpr: max by(node) (max by(instance) (kubelet_running_pod_count{job=\"kubelet\",metrics_path=\"/metrics\"})\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\n  / max by(node) (kube_node_status_capacity_pods{job=\"kube-state-metrics\"} != 1) >\n  0.95\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet '{{ $labels.node }}' is running at {{ $value | humanizePercentage\n    }} of its Pod capacity.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubelettoomanypods\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.712] level=warn ts=2020-07-10T10:55:07.021Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletPodStartUpLatencyHigh\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\"kubelet\",metrics_path=\"/metrics\"}[5m])))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"}\n  > 60\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet Pod startup 99th percentile latency is {{ $value }} seconds on\n    node {{ $labels.node }}.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.713] level=warn ts=2020-07-10T10:55:11.347Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.99\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.714] level=warn ts=2020-07-10T10:55:11.370Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.9, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.9\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.715] level=warn ts=2020-07-10T10:55:11.389Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.5, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.5\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.728] level=warn ts=2020-07-10T10:55:36.940Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletTooManyPods\nexpr: max by(node) (max by(instance) (kubelet_running_pod_count{job=\"kubelet\",metrics_path=\"/metrics\"})\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\n  / max by(node) (kube_node_status_capacity_pods{job=\"kube-state-metrics\"} != 1) >\n  0.95\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet '{{ $labels.node }}' is running at {{ $value | humanizePercentage\n    }} of its Pod capacity.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubelettoomanypods\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.730] level=warn ts=2020-07-10T10:55:36.984Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletPodStartUpLatencyHigh\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\"kubelet\",metrics_path=\"/metrics\"}[5m])))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"}\n  > 60\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet Pod startup 99th percentile latency is {{ $value }} seconds on\n    node {{ $labels.node }}.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.731] level=warn ts=2020-07-10T10:55:41.342Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.99\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.732] level=warn ts=2020-07-10T10:55:41.361Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.9, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.9\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.733] level=warn ts=2020-07-10T10:55:41.383Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.5, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.5\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.734] level=warn ts=2020-07-10T10:56:06.940Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletTooManyPods\nexpr: max by(node) (max by(instance) (kubelet_running_pod_count{job=\"kubelet\",metrics_path=\"/metrics\"})\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\n  / max by(node) (kube_node_status_capacity_pods{job=\"kube-state-metrics\"} != 1) >\n  0.95\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet '{{ $labels.node }}' is running at {{ $value | humanizePercentage\n    }} of its Pod capacity.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubelettoomanypods\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.736] level=warn ts=2020-07-10T10:56:06.984Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletPodStartUpLatencyHigh\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\"kubelet\",metrics_path=\"/metrics\"}[5m])))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"}\n  > 60\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet Pod startup 99th percentile latency is {{ $value }} seconds on\n    node {{ $labels.node }}.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.737] level=warn ts=2020-07-10T10:56:11.339Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.99\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.738] level=warn ts=2020-07-10T10:56:11.355Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.9, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.9\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.739] level=warn ts=2020-07-10T10:56:11.374Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.5, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.5\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.740] level=warn ts=2020-07-10T10:56:36.942Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletTooManyPods\nexpr: max by(node) (max by(instance) (kubelet_running_pod_count{job=\"kubelet\",metrics_path=\"/metrics\"})\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\n  / max by(node) (kube_node_status_capacity_pods{job=\"kube-state-metrics\"} != 1) >\n  0.95\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet '{{ $labels.node }}' is running at {{ $value | humanizePercentage\n    }} of its Pod capacity.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubelettoomanypods\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.741] level=warn ts=2020-07-10T10:56:37.025Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletPodStartUpLatencyHigh\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\"kubelet\",metrics_path=\"/metrics\"}[5m])))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"}\n  > 60\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet Pod startup 99th percentile latency is {{ $value }} seconds on\n    node {{ $labels.node }}.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.742] level=warn ts=2020-07-10T10:56:41.346Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.99\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.743] level=warn ts=2020-07-10T10:56:41.370Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.9, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.9\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.744] level=warn ts=2020-07-10T10:56:41.391Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.5, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.5\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.746] level=warn ts=2020-07-10T10:57:06.942Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletTooManyPods\nexpr: max by(node) (max by(instance) (kubelet_running_pod_count{job=\"kubelet\",metrics_path=\"/metrics\"})\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\n  / max by(node) (kube_node_status_capacity_pods{job=\"kube-state-metrics\"} != 1) >\n  0.95\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet '{{ $labels.node }}' is running at {{ $value | humanizePercentage\n    }} of its Pod capacity.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubelettoomanypods\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.747] level=warn ts=2020-07-10T10:57:07.004Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletPodStartUpLatencyHigh\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\"kubelet\",metrics_path=\"/metrics\"}[5m])))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"}\n  > 60\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet Pod startup 99th percentile latency is {{ $value }} seconds on\n    node {{ $labels.node }}.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.748] level=warn ts=2020-07-10T10:57:11.341Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.99\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.749] level=warn ts=2020-07-10T10:57:11.362Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.9, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.9\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.750] level=warn ts=2020-07-10T10:57:11.381Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.5, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.5\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.751] level=warn ts=2020-07-10T10:57:36.941Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletTooManyPods\nexpr: max by(node) (max by(instance) (kubelet_running_pod_count{job=\"kubelet\",metrics_path=\"/metrics\"})\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\n  / max by(node) (kube_node_status_capacity_pods{job=\"kube-state-metrics\"} != 1) >\n  0.95\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet '{{ $labels.node }}' is running at {{ $value | humanizePercentage\n    }} of its Pod capacity.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubelettoomanypods\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.753] level=warn ts=2020-07-10T10:57:37.016Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletPodStartUpLatencyHigh\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\"kubelet\",metrics_path=\"/metrics\"}[5m])))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"}\n  > 60\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet Pod startup 99th percentile latency is {{ $value }} seconds on\n    node {{ $labels.node }}.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.755] level=warn ts=2020-07-10T10:57:41.348Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.99\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.756] level=warn ts=2020-07-10T10:57:41.369Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.9, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.9\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.757] level=warn ts=2020-07-10T10:57:41.387Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.5, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.5\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.759] level=warn ts=2020-07-10T10:58:06.941Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletTooManyPods\nexpr: max by(node) (max by(instance) (kubelet_running_pod_count{job=\"kubelet\",metrics_path=\"/metrics\"})\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\n  / max by(node) (kube_node_status_capacity_pods{job=\"kube-state-metrics\"} != 1) >\n  0.95\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet '{{ $labels.node }}' is running at {{ $value | humanizePercentage\n    }} of its Pod capacity.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubelettoomanypods\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.760] level=warn ts=2020-07-10T10:58:06.992Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletPodStartUpLatencyHigh\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\"kubelet\",metrics_path=\"/metrics\"}[5m])))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"}\n  > 60\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet Pod startup 99th percentile latency is {{ $value }} seconds on\n    node {{ $labels.node }}.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.761] level=warn ts=2020-07-10T10:58:11.340Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.99\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.762] level=warn ts=2020-07-10T10:58:11.359Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.9, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.9\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.763] level=warn ts=2020-07-10T10:58:11.381Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.5, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.5\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.765] level=warn ts=2020-07-10T10:58:36.942Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletTooManyPods\nexpr: max by(node) (max by(instance) (kubelet_running_pod_count{job=\"kubelet\",metrics_path=\"/metrics\"})\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\n  / max by(node) (kube_node_status_capacity_pods{job=\"kube-state-metrics\"} != 1) >\n  0.95\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet '{{ $labels.node }}' is running at {{ $value | humanizePercentage\n    }} of its Pod capacity.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubelettoomanypods\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.767] level=warn ts=2020-07-10T10:58:37.006Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletPodStartUpLatencyHigh\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\"kubelet\",metrics_path=\"/metrics\"}[5m])))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"}\n  > 60\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet Pod startup 99th percentile latency is {{ $value }} seconds on\n    node {{ $labels.node }}.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.768] level=warn ts=2020-07-10T10:58:41.342Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.99\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.769] level=warn ts=2020-07-10T10:58:41.363Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.9, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.9\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.770] level=warn ts=2020-07-10T10:58:41.382Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.5, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.5\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.771] level=warn ts=2020-07-10T10:59:06.941Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletTooManyPods\nexpr: max by(node) (max by(instance) (kubelet_running_pod_count{job=\"kubelet\",metrics_path=\"/metrics\"})\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\n  / max by(node) (kube_node_status_capacity_pods{job=\"kube-state-metrics\"} != 1) >\n  0.95\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet '{{ $labels.node }}' is running at {{ $value | humanizePercentage\n    }} of its Pod capacity.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubelettoomanypods\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.773] level=warn ts=2020-07-10T10:59:07.004Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletPodStartUpLatencyHigh\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\"kubelet\",metrics_path=\"/metrics\"}[5m])))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"}\n  > 60\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet Pod startup 99th percentile latency is {{ $value }} seconds on\n    node {{ $labels.node }}.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.774] level=warn ts=2020-07-10T10:59:11.345Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.99\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.775] level=warn ts=2020-07-10T10:59:11.366Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.9, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.9\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.776] level=warn ts=2020-07-10T10:59:11.387Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.5, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.5\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.777] level=warn ts=2020-07-10T10:59:36.941Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletTooManyPods\nexpr: max by(node) (max by(instance) (kubelet_running_pod_count{job=\"kubelet\",metrics_path=\"/metrics\"})\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\n  / max by(node) (kube_node_status_capacity_pods{job=\"kube-state-metrics\"} != 1) >\n  0.95\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet '{{ $labels.node }}' is running at {{ $value | humanizePercentage\n    }} of its Pod capacity.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubelettoomanypods\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.778] level=warn ts=2020-07-10T10:59:36.987Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletPodStartUpLatencyHigh\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\"kubelet\",metrics_path=\"/metrics\"}[5m])))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"}\n  > 60\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet Pod startup 99th percentile latency is {{ $value }} seconds on\n    node {{ $labels.node }}.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.779] level=warn ts=2020-07-10T10:59:41.344Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.99\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.780] level=warn ts=2020-07-10T10:59:41.367Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.9, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.9\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.781] level=warn ts=2020-07-10T10:59:41.385Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.5, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.5\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.782] level=warn ts=2020-07-10T11:00:06.941Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletTooManyPods\nexpr: max by(node) (max by(instance) (kubelet_running_pod_count{job=\"kubelet\",metrics_path=\"/metrics\"})\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\n  / max by(node) (kube_node_status_capacity_pods{job=\"kube-state-metrics\"} != 1) >\n  0.95\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet '{{ $labels.node }}' is running at {{ $value | humanizePercentage\n    }} of its Pod capacity.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubelettoomanypods\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.784] level=warn ts=2020-07-10T11:00:06.993Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletPodStartUpLatencyHigh\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\"kubelet\",metrics_path=\"/metrics\"}[5m])))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"}\n  > 60\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet Pod startup 99th percentile latency is {{ $value }} seconds on\n    node {{ $labels.node }}.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.785] level=warn ts=2020-07-10T11:00:11.343Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.99\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.786] level=warn ts=2020-07-10T11:00:11.366Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.9, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.9\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.787] level=warn ts=2020-07-10T11:00:11.388Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.5, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.5\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.788] level=warn ts=2020-07-10T11:00:36.940Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletTooManyPods\nexpr: max by(node) (max by(instance) (kubelet_running_pod_count{job=\"kubelet\",metrics_path=\"/metrics\"})\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\n  / max by(node) (kube_node_status_capacity_pods{job=\"kube-state-metrics\"} != 1) >\n  0.95\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet '{{ $labels.node }}' is running at {{ $value | humanizePercentage\n    }} of its Pod capacity.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubelettoomanypods\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.789] level=warn ts=2020-07-10T11:00:36.990Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletPodStartUpLatencyHigh\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\"kubelet\",metrics_path=\"/metrics\"}[5m])))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"}\n  > 60\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet Pod startup 99th percentile latency is {{ $value }} seconds on\n    node {{ $labels.node }}.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.790] level=warn ts=2020-07-10T11:00:41.346Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.99\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.791] level=warn ts=2020-07-10T11:00:41.370Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.9, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.9\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.792] level=warn ts=2020-07-10T11:00:41.393Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.5, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.5\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.794] level=warn ts=2020-07-10T11:01:06.941Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletTooManyPods\nexpr: max by(node) (max by(instance) (kubelet_running_pod_count{job=\"kubelet\",metrics_path=\"/metrics\"})\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\n  / max by(node) (kube_node_status_capacity_pods{job=\"kube-state-metrics\"} != 1) >\n  0.95\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet '{{ $labels.node }}' is running at {{ $value | humanizePercentage\n    }} of its Pod capacity.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubelettoomanypods\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.795] level=warn ts=2020-07-10T11:01:06.991Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletPodStartUpLatencyHigh\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\"kubelet\",metrics_path=\"/metrics\"}[5m])))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"}\n  > 60\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet Pod startup 99th percentile latency is {{ $value }} seconds on\n    node {{ $labels.node }}.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.796] level=warn ts=2020-07-10T11:01:11.350Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.99\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.797] level=warn ts=2020-07-10T11:01:11.373Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.9, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.9\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.798] level=warn ts=2020-07-10T11:01:11.392Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.5, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.5\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.802] level=warn ts=2020-07-10T11:01:36.942Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletTooManyPods\nexpr: max by(node) (max by(instance) (kubelet_running_pod_count{job=\"kubelet\",metrics_path=\"/metrics\"})\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\n  / max by(node) (kube_node_status_capacity_pods{job=\"kube-state-metrics\"} != 1) >\n  0.95\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet '{{ $labels.node }}' is running at {{ $value | humanizePercentage\n    }} of its Pod capacity.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubelettoomanypods\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.803] level=warn ts=2020-07-10T11:01:37.016Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletPodStartUpLatencyHigh\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\"kubelet\",metrics_path=\"/metrics\"}[5m])))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"}\n  > 60\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet Pod startup 99th percentile latency is {{ $value }} seconds on\n    node {{ $labels.node }}.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.804] level=warn ts=2020-07-10T11:01:41.352Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.99\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.805] level=warn ts=2020-07-10T11:01:41.379Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.9, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.9\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.806] level=warn ts=2020-07-10T11:01:41.401Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.5, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.5\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.808] level=warn ts=2020-07-10T11:02:06.941Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletTooManyPods\nexpr: max by(node) (max by(instance) (kubelet_running_pod_count{job=\"kubelet\",metrics_path=\"/metrics\"})\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\n  / max by(node) (kube_node_status_capacity_pods{job=\"kube-state-metrics\"} != 1) >\n  0.95\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet '{{ $labels.node }}' is running at {{ $value | humanizePercentage\n    }} of its Pod capacity.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubelettoomanypods\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.809] level=warn ts=2020-07-10T11:02:07.022Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletPodStartUpLatencyHigh\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\"kubelet\",metrics_path=\"/metrics\"}[5m])))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"}\n  > 60\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet Pod startup 99th percentile latency is {{ $value }} seconds on\n    node {{ $labels.node }}.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.810] level=warn ts=2020-07-10T11:02:11.348Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.99\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.811] level=warn ts=2020-07-10T11:02:11.373Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.9, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.9\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.812] level=warn ts=2020-07-10T11:02:11.394Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.5, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.5\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.814] level=warn ts=2020-07-10T11:02:36.945Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletTooManyPods\nexpr: max by(node) (max by(instance) (kubelet_running_pod_count{job=\"kubelet\",metrics_path=\"/metrics\"})\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\n  / max by(node) (kube_node_status_capacity_pods{job=\"kube-state-metrics\"} != 1) >\n  0.95\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet '{{ $labels.node }}' is running at {{ $value | humanizePercentage\n    }} of its Pod capacity.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubelettoomanypods\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.815] level=warn ts=2020-07-10T11:02:36.998Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletPodStartUpLatencyHigh\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\"kubelet\",metrics_path=\"/metrics\"}[5m])))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"}\n  > 60\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet Pod startup 99th percentile latency is {{ $value }} seconds on\n    node {{ $labels.node }}.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.816] level=warn ts=2020-07-10T11:02:41.346Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.99\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.817] level=warn ts=2020-07-10T11:02:41.375Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.9, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.9\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.818] level=warn ts=2020-07-10T11:02:41.399Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.5, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.5\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.819] level=warn ts=2020-07-10T11:03:06.940Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletTooManyPods\nexpr: max by(node) (max by(instance) (kubelet_running_pod_count{job=\"kubelet\",metrics_path=\"/metrics\"})\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\n  / max by(node) (kube_node_status_capacity_pods{job=\"kube-state-metrics\"} != 1) >\n  0.95\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet '{{ $labels.node }}' is running at {{ $value | humanizePercentage\n    }} of its Pod capacity.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubelettoomanypods\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.821] level=warn ts=2020-07-10T11:03:06.995Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletPodStartUpLatencyHigh\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\"kubelet\",metrics_path=\"/metrics\"}[5m])))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"}\n  > 60\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet Pod startup 99th percentile latency is {{ $value }} seconds on\n    node {{ $labels.node }}.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.822] level=warn ts=2020-07-10T11:03:11.350Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.99\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.823] level=warn ts=2020-07-10T11:03:11.378Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.9, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.9\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.824] level=warn ts=2020-07-10T11:03:11.403Z caller=manager.go:577 component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.5, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.5\"\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.826] level=warn ts=2020-07-10T11:03:36.942Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletTooManyPods\nexpr: max by(node) (max by(instance) (kubelet_running_pod_count{job=\"kubelet\",metrics_path=\"/metrics\"})\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\n  / max by(node) (kube_node_status_capacity_pods{job=\"kube-state-metrics\"} != 1) >\n  0.95\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet '{{ $labels.node }}' is running at {{ $value | humanizePercentage\n    }} of its Pod capacity.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubelettoomanypods\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.827] level=warn ts=2020-07-10T11:03:37.018Z caller=manager.go:577 component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletPodStartUpLatencyHigh\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\"kubelet\",metrics_path=\"/metrics\"}[5m])))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"}\n  > 60\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: Kubelet Pod startup 99th percentile latency is {{ $value }} seconds on\n    node {{ $labels.node }}.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh\n" err="found duplicate series for the match group {instance=\"10.138.0.59:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-hqwlg2-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.138.0.59:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"gke-charts-testing-1-default-pool-7352761b-gc37\", service=\"prometheus-operator-e0adxo-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
I0710 11:03:40.827] ------------------------------------------------------------------------------------------------------------------------
I0710 11:03:40.827] <== Logs of container prometheus-prometheus-operator-iju343-prometheus-0
I0710 11:03:40.827] ------------------------------------------------------------------------------------------------------------------------
I0710 11:03:40.828] ------------------------------------------------------------------------------------------------------------------------
I0710 11:03:40.828] ==> Logs of container prometheus-prometheus-operator-iju343-prometheus-0
I0710 11:03:40.828] ------------------------------------------------------------------------------------------------------------------------
I0710 11:03:41.093] ts=2020-07-10T10:53:48.068269939Z caller=main.go:85 msg="Starting prometheus-config-reloader version '0.38.1'."
I0710 11:03:41.094] level=error ts=2020-07-10T10:53:48.116339605Z caller=runutil.go:98 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post http://127.0.0.1:9090/-/reload: dial tcp 127.0.0.1:9090: connect: connection refused"
I0710 11:03:41.094] level=info ts=2020-07-10T10:53:53.195491571Z caller=reloader.go:289 msg="Prometheus reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml rule_dirs=
I0710 11:03:41.094] level=info ts=2020-07-10T10:53:53.195602518Z caller=reloader.go:157 msg="started watching config file and non-recursively rule dirs for changes" cfg=/etc/prometheus/config/prometheus.yaml.gz out=/etc/prometheus/config_out/prometheus.env.yaml dirs=
I0710 11:03:41.098] ------------------------------------------------------------------------------------------------------------------------
I0710 11:03:41.098] <== Logs of container prometheus-prometheus-operator-iju343-prometheus-0
I0710 11:03:41.098] ------------------------------------------------------------------------------------------------------------------------
I0710 11:03:41.098] ------------------------------------------------------------------------------------------------------------------------
... skipping 6 lines ...
I0710 11:03:41.400] ========================================================================================================================
I0710 11:03:41.400] Deleting release 'prometheus-operator-iju343rqod'...
I0710 11:03:49.335] release "prometheus-operator-iju343rqod" deleted
I0710 11:03:49.343] Deleting namespace 'metheus-operator-presubmit-23150-1281541524571557888-iju343rqod'...
I0710 11:03:49.619] namespace "metheus-operator-presubmit-23150-1281541524571557888-iju343rqod" deleted
I0710 11:04:01.274] Namespace 'metheus-operator-presubmit-23150-1281541524571557888-iju343rqod' terminated.
I0710 11:04:01.274] Error linting and installing charts: Error processing charts
I0710 11:04:01.274] ------------------------------------------------------------------------------------------------------------------------
I0710 11:04:01.275]  ✖︎ prometheus-operator => (version: "8.16.2", path: "stable/prometheus-operator") > Error waiting for process: exit status 1
I0710 11:04:01.275] ------------------------------------------------------------------------------------------------------------------------
I0710 11:04:03.245] d57c220feaae65f871c551d4d7652cfa4b89be32db624a19dbde63615ab5262c
W0710 11:04:03.276] Traceback (most recent call last):
W0710 11:04:03.277]   File "/workspace/./test-infra/jenkins/../scenarios/execute.py", line 50, in <module>
W0710 11:04:03.282]     main(ARGS.env, ARGS.cmd + ARGS.args)
W0710 11:04:03.282]   File "/workspace/./test-infra/jenkins/../scenarios/execute.py", line 41, in main
W0710 11:04:03.282]     check(*cmd)
W0710 11:04:03.283]   File "/workspace/./test-infra/jenkins/../scenarios/execute.py", line 30, in check
W0710 11:04:03.283]     subprocess.check_call(cmd)
W0710 11:04:03.283]   File "/usr/lib/python2.7/subprocess.py", line 190, in check_call
W0710 11:04:03.283]     raise CalledProcessError(retcode, cmd)
W0710 11:04:03.284] subprocess.CalledProcessError: Command '('./test/e2e.sh',)' returned non-zero exit status 1
E0710 11:04:03.284] Command failed
I0710 11:04:03.284] process 644 exited with code 1 after 12.2m
E0710 11:04:03.285] FAIL: pull-charts-e2e
I0710 11:04:03.285] Call:  gcloud auth activate-service-account --key-file=/etc/service-account/service-account.json
W0710 11:04:04.345] Activated service account credentials for: [pr-kubekins@kubernetes-jenkins-pull.iam.gserviceaccount.com]
I0710 11:04:04.431] process 2033 exited with code 0 after 0.0m
I0710 11:04:04.432] Call:  gcloud config get-value account
I0710 11:04:05.298] process 2047 exited with code 0 after 0.0m
I0710 11:04:05.299] Will upload results to gs://kubernetes-jenkins/pr-logs using pr-kubekins@kubernetes-jenkins-pull.iam.gserviceaccount.com
... skipping 27 lines ...