ci-kubernetes-e2e-capz-master-containerd-windows-2022-serial-slow #1590144996969811968 Results

Test Failures

Kubernetes e2e suite [It] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: CPU) [Serial] [Slow] Deployment (Pod Resource) Should scale from 1 pod to 3 pods and then from 3 pods to 5 pods using Average Utilization for aggregation 15m40s

go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\s\[sig\-autoscaling\]\s\[Feature\:HPA\]\sHorizontal\spod\sautoscaling\s\(scale\sresource\:\sCPU\)\s\[Serial\]\s\[Slow\]\sDeployment\s\(Pod\sResource\)\sShould\sscale\sfrom\s1\spod\sto\s3\spods\sand\sthen\sfrom\s3\spods\sto\s5\spods\susing\sAverage\sUtilization\sfor\saggregation$'

test/e2e/autoscaling/horizontal_pod_autoscaling.go:209
k8s.io/kubernetes/test/e2e/autoscaling.(*HPAScaleTest).run(0xc002041e68, {0x74e3c28?, 0xc00401dbc0?}, {{0x74b92e8, 0x4}, {0x74c2479, 0x7}, {0x74ca886, 0xa}}, 0xc00082cd20)
	test/e2e/autoscaling/horizontal_pod_autoscaling.go:209 +0x2d8
k8s.io/kubernetes/test/e2e/autoscaling.scaleUp({0x74e3c28?, 0x61dc905?}, {{0x74b92e8, 0x4}, {0x74c2479, 0x7}, {0x74ca886, 0xa}}, {0x74b8538, 0x3}, ...)
	test/e2e/autoscaling/horizontal_pod_autoscaling.go:249 +0x212
k8s.io/kubernetes/test/e2e/autoscaling.glob..func6.1.1()
	test/e2e/autoscaling/horizontal_pod_autoscaling.go:50 +0x88
				[BeforeEach] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: CPU)
  set up framework | framework.go:178
STEP: Creating a kubernetes client 11/09/22 02:46:43.98
Nov  9 02:46:43.980: INFO: >>> kubeConfig: /home/prow/go/src/k8s.io/windows-testing/capz-conf-tvyqls.kubeconfig
I1109 02:46:43.982108   14108 discovery.go:214] Invalidating discovery information
STEP: Building a namespace api object, basename horizontal-pod-autoscaling 11/09/22 02:46:43.982
I1109 02:46:44.018953   14108 reflector.go:221] Starting reflector *v1.ServiceAccount (0s) from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
I1109 02:46:44.019154   14108 reflector.go:257] Listing and watching *v1.ServiceAccount from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
I1109 02:46:44.084093   14108 reflector.go:227] Stopping reflector *v1.ServiceAccount (0s) from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
STEP: Waiting for a default service account to be provisioned in namespace 11/09/22 02:46:44.084
I1109 02:46:44.084382   14108 reflector.go:221] Starting reflector *v1.ServiceAccount (0s) from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
I1109 02:46:44.084415   14108 reflector.go:257] Listing and watching *v1.ServiceAccount from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
I1109 02:46:44.145978   14108 reflector.go:227] Stopping reflector *v1.ServiceAccount (0s) from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
STEP: Waiting for kube-root-ca.crt to be provisioned in namespace 11/09/22 02:46:44.146
I1109 02:46:44.146228   14108 reflector.go:221] Starting reflector *v1.ConfigMap (0s) from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
I1109 02:46:44.146236   14108 reflector.go:257] Listing and watching *v1.ConfigMap from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
I1109 02:46:44.208080   14108 reflector.go:227] Stopping reflector *v1.ConfigMap (0s) from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
[BeforeEach] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: CPU)
  test/e2e/framework/metrics/init/init.go:31
[It] Should scale from 1 pod to 3 pods and then from 3 pods to 5 pods using Average Utilization for aggregation
  test/e2e/autoscaling/horizontal_pod_autoscaling.go:49
Nov  9 02:46:44.208: INFO: >>> kubeConfig: /home/prow/go/src/k8s.io/windows-testing/capz-conf-tvyqls.kubeconfig
STEP: Running consuming RC test-deployment via apps/v1beta2, Kind=Deployment with 1 replicas 11/09/22 02:46:44.209
STEP: Creating deployment test-deployment in namespace horizontal-pod-autoscaling-9304 11/09/22 02:46:44.26
I1109 02:46:44.299918   14108 runners.go:193] Created deployment with name: test-deployment, namespace: horizontal-pod-autoscaling-9304, replica count: 1
I1109 02:46:44.300055   14108 reflector.go:221] Starting reflector *v1.Pod (0s) from test/utils/pod_store.go:57
I1109 02:46:44.300070   14108 reflector.go:257] Listing and watching *v1.Pod from test/utils/pod_store.go:57
I1109 02:46:54.350451   14108 runners.go:193] test-deployment Pods: 1 out of 1 created, 1 running, 0 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1109 02:46:54.350914   14108 reflector.go:227] Stopping reflector *v1.Pod (0s) from test/utils/pod_store.go:57
STEP: Running controller 11/09/22 02:46:54.35
STEP: creating replication controller test-deployment-ctrl in namespace horizontal-pod-autoscaling-9304 11/09/22 02:46:54.393
I1109 02:46:54.429292   14108 runners.go:193] Created replication controller with name: test-deployment-ctrl, namespace: horizontal-pod-autoscaling-9304, replica count: 1
I1109 02:46:54.429434   14108 reflector.go:221] Starting reflector *v1.Pod (0s) from test/utils/pod_store.go:57
I1109 02:46:54.429447   14108 reflector.go:257] Listing and watching *v1.Pod from test/utils/pod_store.go:57
I1109 02:47:04.482550   14108 runners.go:193] test-deployment-ctrl Pods: 1 out of 1 created, 1 running, 0 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1109 02:47:04.482845   14108 reflector.go:227] Stopping reflector *v1.Pod (0s) from test/utils/pod_store.go:57
Nov  9 02:47:09.486: INFO: Waiting for amount of service:test-deployment-ctrl endpoints to be 1
Nov  9 02:47:09.518: INFO: RC test-deployment: consume 250 millicores in total
Nov  9 02:47:09.518: INFO: RC test-deployment: setting consumption to 250 millicores in total
Nov  9 02:47:09.518: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 02:47:09.518: INFO: RC test-deployment: consume 0 MB in total
Nov  9 02:47:09.518: INFO: RC test-deployment: disabling mem consumption
Nov  9 02:47:09.518: INFO: RC test-deployment: consume custom metric 0 in total
Nov  9 02:47:09.518: INFO: RC test-deployment: disabling consumption of custom metric QPS
Nov  9 02:47:09.518: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-9304/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:47:09.587: INFO: waiting for 3 replicas (current: 1)
Nov  9 02:47:29.625: INFO: waiting for 3 replicas (current: 1)
Nov  9 02:47:39.581: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 02:47:39.581: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-9304/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:47:49.622: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:48:09.621: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:48:09.632: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 02:48:09.632: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-9304/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:48:29.624: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:48:39.684: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 02:48:39.684: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-9304/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:48:49.622: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:49:09.621: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:49:09.750: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 02:49:09.750: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-9304/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:49:29.621: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:49:39.795: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 02:49:39.795: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-9304/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:49:49.627: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:50:09.621: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:50:09.840: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 02:50:09.841: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-9304/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:50:29.624: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:50:39.891: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 02:50:39.891: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-9304/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:50:49.623: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:51:09.621: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:51:09.939: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 02:51:09.939: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-9304/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:51:29.624: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:51:39.984: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 02:51:39.985: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-9304/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:51:49.621: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:52:09.622: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:52:10.029: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 02:52:10.029: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-9304/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:52:29.621: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:52:40.078: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 02:52:40.078: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-9304/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:52:49.624: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:53:09.621: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:53:10.123: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 02:53:10.123: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-9304/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:53:29.623: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:53:40.174: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 02:53:40.174: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-9304/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:53:49.621: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:54:09.621: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:54:10.225: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 02:54:10.225: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-9304/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:54:29.624: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:54:40.271: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 02:54:40.271: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-9304/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:54:49.622: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:55:09.622: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:55:10.324: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 02:55:10.325: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-9304/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:55:29.623: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:55:40.379: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 02:55:40.379: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-9304/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:55:49.622: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:56:09.621: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:56:10.424: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 02:56:10.424: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-9304/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:56:29.625: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:56:40.470: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 02:56:40.470: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-9304/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:56:49.621: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:57:09.621: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:57:10.515: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 02:57:10.515: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-9304/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:57:29.624: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:57:40.560: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 02:57:40.560: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-9304/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:57:49.624: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:58:09.622: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:58:10.606: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 02:58:10.606: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-9304/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:58:29.624: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:58:40.653: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 02:58:40.653: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-9304/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:58:49.627: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:59:09.622: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:59:10.700: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 02:59:10.700: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-9304/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:59:29.621: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:59:40.744: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 02:59:40.744: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-9304/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:59:49.621: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:00:09.625: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:00:10.790: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 03:00:10.790: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-9304/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 03:00:29.624: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:00:40.836: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 03:00:40.836: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-9304/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 03:00:49.625: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:01:09.622: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:01:10.882: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 03:01:10.882: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-9304/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 03:01:29.621: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:01:40.928: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 03:01:40.928: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-9304/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 03:01:49.621: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:02:09.621: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:02:09.655: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:02:09.655: INFO: Unexpected error: timeout waiting 15m0s for 3 replicas: 
    <*errors.errorString | 0xc000287c80>: {
        s: "timed out waiting for the condition",
    }
Nov  9 03:02:09.655: FAIL: timeout waiting 15m0s for 3 replicas: timed out waiting for the condition

Full Stack Trace
k8s.io/kubernetes/test/e2e/autoscaling.(*HPAScaleTest).run(0xc002041e68, {0x74e3c28?, 0xc00401dbc0?}, {{0x74b92e8, 0x4}, {0x74c2479, 0x7}, {0x74ca886, 0xa}}, 0xc00082cd20)
	test/e2e/autoscaling/horizontal_pod_autoscaling.go:209 +0x2d8
k8s.io/kubernetes/test/e2e/autoscaling.scaleUp({0x74e3c28?, 0x61dc905?}, {{0x74b92e8, 0x4}, {0x74c2479, 0x7}, {0x74ca886, 0xa}}, {0x74b8538, 0x3}, ...)
	test/e2e/autoscaling/horizontal_pod_autoscaling.go:249 +0x212
k8s.io/kubernetes/test/e2e/autoscaling.glob..func6.1.1()
	test/e2e/autoscaling/horizontal_pod_autoscaling.go:50 +0x88
STEP: Removing consuming RC test-deployment 11/09/22 03:02:09.691
Nov  9 03:02:09.691: INFO: RC test-deployment: stopping metric consumer
Nov  9 03:02:09.691: INFO: RC test-deployment: stopping CPU consumer
Nov  9 03:02:09.691: INFO: RC test-deployment: stopping mem consumer
STEP: deleting Deployment.apps test-deployment in namespace horizontal-pod-autoscaling-9304, will wait for the garbage collector to delete the pods 11/09/22 03:02:19.691
I1109 03:02:19.725109   14108 reflector.go:221] Starting reflector *v1.Pod (0s) from test/utils/pod_store.go:57
I1109 03:02:19.725145   14108 reflector.go:257] Listing and watching *v1.Pod from test/utils/pod_store.go:57
Nov  9 03:02:19.862: INFO: Deleting Deployment.apps test-deployment took: 36.530615ms
I1109 03:02:19.963459   14108 resource.go:655] Ignoring inactive pod horizontal-pod-autoscaling-9304/test-deployment-669bb6996d-vmwln in state Running, deletion time 2022-11-09 03:02:49 +0000 UTC
I1109 03:02:19.963525   14108 resource.go:655] Ignoring inactive pod horizontal-pod-autoscaling-9304/test-deployment-669bb6996d-tbvnd in state Running, deletion time 2022-11-09 03:02:49 +0000 UTC
Nov  9 03:02:19.963: INFO: Terminating Deployment.apps test-deployment pods took: 100.963764ms
I1109 03:02:21.764474   14108 reflector.go:227] Stopping reflector *v1.Pod (0s) from test/utils/pod_store.go:57
STEP: deleting ReplicationController test-deployment-ctrl in namespace horizontal-pod-autoscaling-9304, will wait for the garbage collector to delete the pods 11/09/22 03:02:21.827
I1109 03:02:21.859737   14108 reflector.go:221] Starting reflector *v1.Pod (0s) from test/utils/pod_store.go:57
I1109 03:02:21.859904   14108 reflector.go:257] Listing and watching *v1.Pod from test/utils/pod_store.go:57
Nov  9 03:02:21.945: INFO: Deleting ReplicationController test-deployment-ctrl took: 35.134505ms
I1109 03:02:22.046364   14108 resource.go:655] Ignoring inactive pod horizontal-pod-autoscaling-9304/test-deployment-ctrl-zdk8k in state Running, deletion time 2022-11-09 03:02:22 +0000 UTC
Nov  9 03:02:22.046: INFO: Terminating ReplicationController test-deployment-ctrl pods took: 101.250975ms
I1109 03:02:23.747681   14108 reflector.go:227] Stopping reflector *v1.Pod (0s) from test/utils/pod_store.go:57
[AfterEach] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: CPU)
  test/e2e/framework/node/init/init.go:32
Nov  9 03:02:23.811: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
[DeferCleanup (Each)] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: CPU)
  test/e2e/framework/metrics/init/init.go:33
[DeferCleanup (Each)] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: CPU)
  dump namespaces | framework.go:196
STEP: dump namespace information after failure 11/09/22 03:02:23.846
STEP: Collecting events from namespace "horizontal-pod-autoscaling-9304". 11/09/22 03:02:23.847
STEP: Found 22 events. 11/09/22 03:02:23.883
Nov  9 03:02:23.883: INFO: At 2022-11-09 02:46:44 +0000 UTC - event for test-deployment: {deployment-controller } ScalingReplicaSet: Scaled up replica set test-deployment-669bb6996d to 1
Nov  9 03:02:23.883: INFO: At 2022-11-09 02:46:44 +0000 UTC - event for test-deployment-669bb6996d: {replicaset-controller } SuccessfulCreate: Created pod: test-deployment-669bb6996d-vmwln
Nov  9 03:02:23.883: INFO: At 2022-11-09 02:46:44 +0000 UTC - event for test-deployment-669bb6996d-vmwln: {default-scheduler } Scheduled: Successfully assigned horizontal-pod-autoscaling-9304/test-deployment-669bb6996d-vmwln to capz-conf-m97cx
Nov  9 03:02:23.883: INFO: At 2022-11-09 02:46:45 +0000 UTC - event for test-deployment-669bb6996d-vmwln: {kubelet capz-conf-m97cx} FailedMount: MountVolume.SetUp failed for volume "kube-api-access-jqcl6" : failed to sync configmap cache: timed out waiting for the condition
Nov  9 03:02:23.883: INFO: At 2022-11-09 02:46:49 +0000 UTC - event for test-deployment-669bb6996d-vmwln: {kubelet capz-conf-m97cx} Pulled: Container image "registry.k8s.io/e2e-test-images/resource-consumer:1.13" already present on machine
Nov  9 03:02:23.883: INFO: At 2022-11-09 02:46:49 +0000 UTC - event for test-deployment-669bb6996d-vmwln: {kubelet capz-conf-m97cx} Created: Created container test-deployment
Nov  9 03:02:23.883: INFO: At 2022-11-09 02:46:52 +0000 UTC - event for test-deployment-669bb6996d-vmwln: {kubelet capz-conf-m97cx} Started: Started container test-deployment
Nov  9 03:02:23.883: INFO: At 2022-11-09 02:46:54 +0000 UTC - event for test-deployment-ctrl: {replication-controller } SuccessfulCreate: Created pod: test-deployment-ctrl-zdk8k
Nov  9 03:02:23.883: INFO: At 2022-11-09 02:46:54 +0000 UTC - event for test-deployment-ctrl-zdk8k: {default-scheduler } Scheduled: Successfully assigned horizontal-pod-autoscaling-9304/test-deployment-ctrl-zdk8k to capz-conf-hhzhh
Nov  9 03:02:23.883: INFO: At 2022-11-09 02:46:58 +0000 UTC - event for test-deployment-ctrl-zdk8k: {kubelet capz-conf-hhzhh} Created: Created container test-deployment-ctrl
Nov  9 03:02:23.883: INFO: At 2022-11-09 02:46:58 +0000 UTC - event for test-deployment-ctrl-zdk8k: {kubelet capz-conf-hhzhh} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.40" already present on machine
Nov  9 03:02:23.883: INFO: At 2022-11-09 02:47:01 +0000 UTC - event for test-deployment-ctrl-zdk8k: {kubelet capz-conf-hhzhh} Started: Started container test-deployment-ctrl
Nov  9 03:02:23.883: INFO: At 2022-11-09 02:47:24 +0000 UTC - event for test-deployment: {horizontal-pod-autoscaler } SuccessfulRescale: New size: 2; reason: cpu resource utilization (percentage of request) above target
Nov  9 03:02:23.883: INFO: At 2022-11-09 02:47:24 +0000 UTC - event for test-deployment: {deployment-controller } ScalingReplicaSet: Scaled up replica set test-deployment-669bb6996d to 2 from 1
Nov  9 03:02:23.883: INFO: At 2022-11-09 02:47:24 +0000 UTC - event for test-deployment-669bb6996d: {replicaset-controller } SuccessfulCreate: Created pod: test-deployment-669bb6996d-tbvnd
Nov  9 03:02:23.883: INFO: At 2022-11-09 02:47:24 +0000 UTC - event for test-deployment-669bb6996d-tbvnd: {default-scheduler } Scheduled: Successfully assigned horizontal-pod-autoscaling-9304/test-deployment-669bb6996d-tbvnd to capz-conf-hhzhh
Nov  9 03:02:23.883: INFO: At 2022-11-09 02:47:28 +0000 UTC - event for test-deployment-669bb6996d-tbvnd: {kubelet capz-conf-hhzhh} Created: Created container test-deployment
Nov  9 03:02:23.883: INFO: At 2022-11-09 02:47:28 +0000 UTC - event for test-deployment-669bb6996d-tbvnd: {kubelet capz-conf-hhzhh} Pulled: Container image "registry.k8s.io/e2e-test-images/resource-consumer:1.13" already present on machine
Nov  9 03:02:23.883: INFO: At 2022-11-09 02:47:31 +0000 UTC - event for test-deployment-669bb6996d-tbvnd: {kubelet capz-conf-hhzhh} Started: Started container test-deployment
Nov  9 03:02:23.883: INFO: At 2022-11-09 03:02:19 +0000 UTC - event for test-deployment-669bb6996d-tbvnd: {kubelet capz-conf-hhzhh} Killing: Stopping container test-deployment
Nov  9 03:02:23.883: INFO: At 2022-11-09 03:02:19 +0000 UTC - event for test-deployment-669bb6996d-vmwln: {kubelet capz-conf-m97cx} Killing: Stopping container test-deployment
Nov  9 03:02:23.883: INFO: At 2022-11-09 03:02:21 +0000 UTC - event for test-deployment-ctrl-zdk8k: {kubelet capz-conf-hhzhh} Killing: Stopping container test-deployment-ctrl
Nov  9 03:02:23.915: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
Nov  9 03:02:23.915: INFO: 
Nov  9 03:02:23.950: INFO: 
Logging node info for node capz-conf-hhzhh
Nov  9 03:02:23.983: INFO: Node Info: &Node{ObjectMeta:{capz-conf-hhzhh    641b6838-eeea-48b0-9dc0-0ff9ab096f10 16461 0 2022-11-09 01:07:16 +0000 UTC <nil> <nil> map[agentpool:windowsgmsa beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:windows failure-domain.beta.kubernetes.io/region:eastus failure-domain.beta.kubernetes.io/zone:0 kubernetes.io/arch:amd64 kubernetes.io/hostname:capz-conf-hhzhh kubernetes.io/os:windows node.kubernetes.io/instance-type:Standard_D4s_v3 node.kubernetes.io/windows-build:10.0.20348 topology.kubernetes.io/region:eastus topology.kubernetes.io/zone:0] map[cluster.x-k8s.io/cluster-name:capz-conf-tvyqls cluster.x-k8s.io/cluster-namespace:default cluster.x-k8s.io/machine:capz-conf-tvyqls-md-win-7848cfdb84-9mln2 cluster.x-k8s.io/owner-kind:MachineSet cluster.x-k8s.io/owner-name:capz-conf-tvyqls-md-win-7848cfdb84 kubeadm.alpha.kubernetes.io/cri-socket:npipe:////./pipe/containerd-containerd node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:10.1.0.4/16 projectcalico.org/IPv4VXLANTunnelAddr:192.168.84.65 projectcalico.org/VXLANTunnelMACAddr:00:15:5d:5b:b6:ce volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubeadm Update v1 2022-11-09 01:07:16 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:kubeadm.alpha.kubernetes.io/cri-socket":{}}}} } {kubelet.exe Update v1 2022-11-09 01:07:16 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:node.kubernetes.io/windows-build":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {manager Update v1 2022-11-09 01:08:02 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:cluster.x-k8s.io/cluster-name":{},"f:cluster.x-k8s.io/cluster-namespace":{},"f:cluster.x-k8s.io/machine":{},"f:cluster.x-k8s.io/owner-kind":{},"f:cluster.x-k8s.io/owner-name":{}}}} } {kube-controller-manager Update v1 2022-11-09 01:08:06 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}}} } {Go-http-client Update v1 2022-11-09 01:08:32 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4VXLANTunnelAddr":{},"f:projectcalico.org/VXLANTunnelMACAddr":{}}}} status} {cluster-api-e2e Update v1 2022-11-09 01:10:08 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:agentpool":{}}}} } {e2e.test Update v1 2022-11-09 01:22:08 +0000 UTC FieldsV1 {"f:status":{"f:capacity":{"f:example.com/fakePTSRes":{}}}} status} {kubelet.exe Update v1 2022-11-09 02:59:14 +0000 UTC FieldsV1 {"f:status":{"f:allocatable":{"f:example.com/fakePTSRes":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:,DoNotUseExternalID:,ProviderID:azure:///subscriptions/===REDACTED===/resourceGroups/capz-conf-tvyqls/providers/Microsoft.Compute/virtualMachines/capz-conf-hhzhh,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-azure-disk: {{8 0} {<nil>} 8 DecimalSI},cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{136912564224 0} {<nil>} 133703676Ki BinarySI},example.com/fakePTSRes: {{10 0} {<nil>} 10 DecimalSI},memory: {{17179398144 0} {<nil>} 16776756Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-azure-disk: {{8 0} {<nil>} 8 DecimalSI},cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{123221307598 0} {<nil>} 123221307598 DecimalSI},example.com/fakePTSRes: {{10 0} {<nil>} 10 DecimalSI},memory: {{17074540544 0} {<nil>} 16674356Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-09 02:59:14 +0000 UTC,LastTransitionTime:2022-11-09 01:07:16 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-09 02:59:14 +0000 UTC,LastTransitionTime:2022-11-09 01:07:16 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-09 02:59:14 +0000 UTC,LastTransitionTime:2022-11-09 01:07:16 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-09 02:59:14 +0000 UTC,LastTransitionTime:2022-11-09 01:08:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status,},},Addresses:[]NodeAddress{NodeAddress{Type:Hostname,Address:capz-conf-hhzhh,},NodeAddress{Type:InternalIP,Address:10.1.0.4,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:capz-conf-hhzhh,SystemUUID:5F9EA3E7-22F1-406E-9AAC-69BA5DD83BF4,BootID:8,KernelVersion:10.0.20348.1129,OSImage:Windows Server 2022 Datacenter,ContainerRuntimeVersion:containerd://1.6.8,KubeletVersion:v1.26.0-alpha.3.425+2f2240400391ad,KubeProxyVersion:v1.26.0-alpha.3.425+2f2240400391ad,OperatingSystem:windows,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:307641217,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:221349084,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:218942479,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:183620591,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:181784786,},ContainerImage{Names:[ghcr.io/kubernetes-sigs/sig-windows/eventflow-logger@sha256:63bf2aa9db909d0d90fb5205abf7fb2a6d9a494b89cbd2508a42457dfc875505 ghcr.io/kubernetes-sigs/sig-windows/eventflow-logger:v0.1.0],SizeBytes:148054279,},ContainerImage{Names:[ghcr.io/kubernetes-sigs/sig-windows/csi-proxy@sha256:96b4144986319a747ba599892454be2737aae6005d96b8e13ed481321ac3afba ghcr.io/kubernetes-sigs/sig-windows/csi-proxy:v1.0.2],SizeBytes:123925937,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/resource-consumer@sha256:ba5e047a337e5d0709bc57df45b95b2c7f6f2794b290e4e24f7fc8980d60b25a registry.k8s.io/e2e-test-images/resource-consumer:1.13],SizeBytes:121182992,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6],SizeBytes:119397646,},ContainerImage{Names:[docker.io/sigwindowstools/kube-proxy:v1.23.1-calico-hostprocess docker.io/sigwindowstools/kube-proxy:v1.26.0-alpha.3.425_2f2240400391ad-calico-hostprocess],SizeBytes:116182072,},ContainerImage{Names:[docker.io/sigwindowstools/calico-install@sha256:97bc10aa5000a0ee1c842ac32771fe7a45a3a5ca507711bdf57ae2eb5f293e2b docker.io/sigwindowstools/calico-install:v3.23.0-hostprocess],SizeBytes:47258343,},ContainerImage{Names:[docker.io/sigwindowstools/calico-node@sha256:ea8b55bde9aed6a649582a6e21029577430661c743d94b3a5e93d57e648874a2 docker.io/sigwindowstools/calico-node:v3.23.0-hostprocess],SizeBytes:27005624,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Nov  9 03:02:23.983: INFO: 
Logging kubelet events for node capz-conf-hhzhh
Nov  9 03:02:24.015: INFO: 
Logging pods the kubelet thinks is on node capz-conf-hhzhh
Nov  9 03:02:24.066: INFO: containerd-logger-z56l7 started at 2022-11-09 01:07:16 +0000 UTC (0+1 container statuses recorded)
Nov  9 03:02:24.067: INFO: 	Container containerd-logger ready: true, restart count 0
Nov  9 03:02:24.067: INFO: kube-proxy-windows-ldnh5 started at 2022-11-09 01:07:16 +0000 UTC (0+1 container statuses recorded)
Nov  9 03:02:24.067: INFO: 	Container kube-proxy ready: true, restart count 0
Nov  9 03:02:24.067: INFO: calico-node-windows-jpm48 started at 2022-11-09 01:07:16 +0000 UTC (1+2 container statuses recorded)
Nov  9 03:02:24.067: INFO: 	Init container install-cni ready: true, restart count 0
Nov  9 03:02:24.067: INFO: 	Container calico-node-felix ready: true, restart count 1
Nov  9 03:02:24.067: INFO: 	Container calico-node-startup ready: true, restart count 0
Nov  9 03:02:24.067: INFO: csi-proxy-dkn8s started at 2022-11-09 01:08:06 +0000 UTC (0+1 container statuses recorded)
Nov  9 03:02:24.067: INFO: 	Container csi-proxy ready: true, restart count 0
Nov  9 03:02:24.247: INFO: 
Latency metrics for node capz-conf-hhzhh
Nov  9 03:02:24.248: INFO: 
Logging node info for node capz-conf-m97cx
Nov  9 03:02:24.281: INFO: Node Info: &Node{ObjectMeta:{capz-conf-m97cx    36965107-96dc-426b-924d-c291c94a2d04 16449 0 2022-11-09 01:07:00 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:windows failure-domain.beta.kubernetes.io/region:eastus failure-domain.beta.kubernetes.io/zone:0 kubernetes.io/arch:amd64 kubernetes.io/hostname:capz-conf-m97cx kubernetes.io/os:windows node.kubernetes.io/instance-type:Standard_D4s_v3 node.kubernetes.io/windows-build:10.0.20348 topology.kubernetes.io/region:eastus topology.kubernetes.io/zone:0] map[cluster.x-k8s.io/cluster-name:capz-conf-tvyqls cluster.x-k8s.io/cluster-namespace:default cluster.x-k8s.io/machine:capz-conf-tvyqls-md-win-7848cfdb84-cw8bk cluster.x-k8s.io/owner-kind:MachineSet cluster.x-k8s.io/owner-name:capz-conf-tvyqls-md-win-7848cfdb84 kubeadm.alpha.kubernetes.io/cri-socket:npipe:////./pipe/containerd-containerd node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:10.1.0.5/16 projectcalico.org/IPv4VXLANTunnelAddr:192.168.100.1 projectcalico.org/VXLANTunnelMACAddr:00:15:5d:ac:92:68 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubeadm Update v1 2022-11-09 01:07:00 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:kubeadm.alpha.kubernetes.io/cri-socket":{}}}} } {kubelet.exe Update v1 2022-11-09 01:07:00 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:node.kubernetes.io/windows-build":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-11-09 01:07:31 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}}} } {manager Update v1 2022-11-09 01:08:02 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:cluster.x-k8s.io/cluster-name":{},"f:cluster.x-k8s.io/cluster-namespace":{},"f:cluster.x-k8s.io/machine":{},"f:cluster.x-k8s.io/owner-kind":{},"f:cluster.x-k8s.io/owner-name":{}}}} } {Go-http-client Update v1 2022-11-09 01:08:08 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4VXLANTunnelAddr":{},"f:projectcalico.org/VXLANTunnelMACAddr":{}}}} status} {e2e.test Update v1 2022-11-09 01:22:07 +0000 UTC FieldsV1 {"f:status":{"f:capacity":{"f:example.com/fakePTSRes":{}}}} status} {kubelet.exe Update v1 2022-11-09 02:59:08 +0000 UTC FieldsV1 {"f:status":{"f:allocatable":{"f:example.com/fakePTSRes":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:,DoNotUseExternalID:,ProviderID:azure:///subscriptions/===REDACTED===/resourceGroups/capz-conf-tvyqls/providers/Microsoft.Compute/virtualMachines/capz-conf-m97cx,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-azure-disk: {{8 0} {<nil>} 8 DecimalSI},cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{136912564224 0} {<nil>} 133703676Ki BinarySI},example.com/fakePTSRes: {{10 0} {<nil>} 10 DecimalSI},memory: {{17179398144 0} {<nil>} 16776756Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-azure-disk: {{8 0} {<nil>} 8 DecimalSI},cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{123221307598 0} {<nil>} 123221307598 DecimalSI},example.com/fakePTSRes: {{10 0} {<nil>} 10 DecimalSI},memory: {{17074540544 0} {<nil>} 16674356Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-09 02:59:08 +0000 UTC,LastTransitionTime:2022-11-09 01:07:00 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-09 02:59:08 +0000 UTC,LastTransitionTime:2022-11-09 01:07:00 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-09 02:59:08 +0000 UTC,LastTransitionTime:2022-11-09 01:07:00 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-09 02:59:08 +0000 UTC,LastTransitionTime:2022-11-09 01:07:31 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status,},},Addresses:[]NodeAddress{NodeAddress{Type:Hostname,Address:capz-conf-m97cx,},NodeAddress{Type:InternalIP,Address:10.1.0.5,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:capz-conf-m97cx,SystemUUID:FC504DDA-1329-4BBA-8C1E-77FAD783BCA1,BootID:8,KernelVersion:10.0.20348.1129,OSImage:Windows Server 2022 Datacenter,ContainerRuntimeVersion:containerd://1.6.8,KubeletVersion:v1.26.0-alpha.3.425+2f2240400391ad,KubeProxyVersion:v1.26.0-alpha.3.425+2f2240400391ad,OperatingSystem:windows,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:307641217,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:221349084,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:218942479,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:183620591,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:181784786,},ContainerImage{Names:[ghcr.io/kubernetes-sigs/sig-windows/eventflow-logger@sha256:63bf2aa9db909d0d90fb5205abf7fb2a6d9a494b89cbd2508a42457dfc875505 ghcr.io/kubernetes-sigs/sig-windows/eventflow-logger:v0.1.0],SizeBytes:148054279,},ContainerImage{Names:[ghcr.io/kubernetes-sigs/sig-windows/csi-proxy@sha256:96b4144986319a747ba599892454be2737aae6005d96b8e13ed481321ac3afba ghcr.io/kubernetes-sigs/sig-windows/csi-proxy:v1.0.2],SizeBytes:123925937,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/resource-consumer@sha256:ba5e047a337e5d0709bc57df45b95b2c7f6f2794b290e4e24f7fc8980d60b25a registry.k8s.io/e2e-test-images/resource-consumer:1.13],SizeBytes:121182992,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6],SizeBytes:119397646,},ContainerImage{Names:[docker.io/sigwindowstools/kube-proxy:v1.23.1-calico-hostprocess docker.io/sigwindowstools/kube-proxy:v1.26.0-alpha.3.425_2f2240400391ad-calico-hostprocess],SizeBytes:116182072,},ContainerImage{Names:[docker.io/sigwindowstools/calico-install@sha256:97bc10aa5000a0ee1c842ac32771fe7a45a3a5ca507711bdf57ae2eb5f293e2b docker.io/sigwindowstools/calico-install:v3.23.0-hostprocess],SizeBytes:47258343,},ContainerImage{Names:[docker.io/sigwindowstools/calico-node@sha256:ea8b55bde9aed6a649582a6e21029577430661c743d94b3a5e93d57e648874a2 docker.io/sigwindowstools/calico-node:v3.23.0-hostprocess],SizeBytes:27005624,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Nov  9 03:02:24.281: INFO: 
Logging kubelet events for node capz-conf-m97cx
Nov  9 03:02:24.313: INFO: 
Logging pods the kubelet thinks is on node capz-conf-m97cx
Nov  9 03:02:24.367: INFO: calico-node-windows-w5q9w started at 2022-11-09 01:07:01 +0000 UTC (1+2 container statuses recorded)
Nov  9 03:02:24.367: INFO: 	Init container install-cni ready: true, restart count 0
Nov  9 03:02:24.367: INFO: 	Container calico-node-felix ready: true, restart count 1
Nov  9 03:02:24.367: INFO: 	Container calico-node-startup ready: true, restart count 0
Nov  9 03:02:24.367: INFO: containerd-logger-v9qh5 started at 2022-11-09 01:07:01 +0000 UTC (0+1 container statuses recorded)
Nov  9 03:02:24.367: INFO: 	Container containerd-logger ready: true, restart count 0
Nov  9 03:02:24.367: INFO: kube-proxy-windows-55jdx started at 2022-11-09 01:07:01 +0000 UTC (0+1 container statuses recorded)
Nov  9 03:02:24.367: INFO: 	Container kube-proxy ready: true, restart count 0
Nov  9 03:02:24.367: INFO: csi-proxy-tn9x7 started at 2022-11-09 01:07:31 +0000 UTC (0+1 container statuses recorded)
Nov  9 03:02:24.367: INFO: 	Container csi-proxy ready: true, restart count 0
Nov  9 03:02:24.535: INFO: 
Latency metrics for node capz-conf-m97cx
Nov  9 03:02:24.535: INFO: 
Logging node info for node capz-conf-tvyqls-control-plane-7vbm8
Nov  9 03:02:24.567: INFO: Node Info: &Node{ObjectMeta:{capz-conf-tvyqls-control-plane-7vbm8    59067037-dec8-4e79-8a8a-d150ea7ce6ba 16518 0 2022-11-09 01:04:38 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D2s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eastus failure-domain.beta.kubernetes.io/zone:eastus-3 kubernetes.io/arch:amd64 kubernetes.io/hostname:capz-conf-tvyqls-control-plane-7vbm8 kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:Standard_D2s_v3 topology.kubernetes.io/region:eastus topology.kubernetes.io/zone:eastus-3] map[cluster.x-k8s.io/cluster-name:capz-conf-tvyqls cluster.x-k8s.io/cluster-namespace:default cluster.x-k8s.io/machine:capz-conf-tvyqls-control-plane-ljqrh cluster.x-k8s.io/owner-kind:KubeadmControlPlane cluster.x-k8s.io/owner-name:capz-conf-tvyqls-control-plane kubeadm.alpha.kubernetes.io/cri-socket:unix:///var/run/containerd/containerd.sock node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:10.0.0.4/16 projectcalico.org/IPv4VXLANTunnelAddr:192.168.2.192 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-11-09 01:04:38 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kubeadm Update v1 2022-11-09 01:04:40 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:kubeadm.alpha.kubernetes.io/cri-socket":{}},"f:labels":{"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {manager Update v1 2022-11-09 01:04:55 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:cluster.x-k8s.io/cluster-name":{},"f:cluster.x-k8s.io/cluster-namespace":{},"f:cluster.x-k8s.io/machine":{},"f:cluster.x-k8s.io/owner-kind":{},"f:cluster.x-k8s.io/owner-name":{}}}} } {kube-controller-manager Update v1 2022-11-09 01:05:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:taints":{}}} } {Go-http-client Update v1 2022-11-09 01:05:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4VXLANTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-09 02:59:51 +0000 UTC FieldsV1 {"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:,DoNotUseExternalID:,ProviderID:azure:///subscriptions/===REDACTED===/resourceGroups/capz-conf-tvyqls/providers/Microsoft.Compute/virtualMachines/capz-conf-tvyqls-control-plane-7vbm8,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-azure-disk: {{4 0} {<nil>} 4 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{133003395072 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8344723456 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-azure-disk: {{4 0} {<nil>} 4 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{119703055367 0} {<nil>} 119703055367 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8239865856 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-09 01:05:27 +0000 UTC,LastTransitionTime:2022-11-09 01:05:27 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-09 02:59:51 +0000 UTC,LastTransitionTime:2022-11-09 01:04:17 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-09 02:59:51 +0000 UTC,LastTransitionTime:2022-11-09 01:04:17 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-09 02:59:51 +0000 UTC,LastTransitionTime:2022-11-09 01:04:17 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-09 02:59:51 +0000 UTC,LastTransitionTime:2022-11-09 01:05:22 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:Hostname,Address:capz-conf-tvyqls-control-plane-7vbm8,},NodeAddress{Type:InternalIP,Address:10.0.0.4,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:37a2e1ceaa0d455a8de0dda7f144eb7f,SystemUUID:50325a86-a564-f642-b7cf-2b4ce49e7328,BootID:49adcb05-103e-4672-b80c-d3eea5cf97ca,KernelVersion:5.4.0-1091-azure,OSImage:Ubuntu 18.04.6 LTS,ContainerRuntimeVersion:containerd://1.6.2,KubeletVersion:v1.26.0-alpha.3.425+2f2240400391ad,KubeProxyVersion:v1.26.0-alpha.3.425+2f2240400391ad,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[docker.io/calico/cni@sha256:914823d144204288f881e49b93b6852febfe669074cd4e2a782860981615f521 docker.io/calico/cni:v3.23.0],SizeBytes:110494683,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:b83c1d70989e1fe87583607bf5aee1ee34e52773d4755b95f5cf5a451962f3a4 registry.k8s.io/etcd:3.5.5-0],SizeBytes:102417044,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:102157811,},ContainerImage{Names:[docker.io/calico/node@sha256:4763820ecb4d8e82483a2ffabfec7fcded9603318692df210a778d223a4d7474 docker.io/calico/node:v3.23.0],SizeBytes:71573794,},ContainerImage{Names:[docker.io/calico/kube-controllers@sha256:78bc199299f966b0694dc4044501aee2d7ebd6862b2b0a00bca3ee8d3813c82f docker.io/calico/kube-controllers:v3.23.0],SizeBytes:56343954,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:51155161,},ContainerImage{Names:[gcr.io/k8s-staging-ci-images/kube-apiserver@sha256:f4a034110c7fecfbd53047bb4e7a568b1d123694426129781f718b2b83f650dd gcr.io/k8s-staging-ci-images/kube-apiserver:v1.26.0-alpha.3.425_2f2240400391ad],SizeBytes:34983033,},ContainerImage{Names:[registry.k8s.io/kube-apiserver@sha256:4188262a351f156e8027ff81693d771c35b34b668cbd61e59c4a4490dd5c08f3 registry.k8s.io/kube-apiserver:v1.25.3],SizeBytes:34238163,},ContainerImage{Names:[gcr.io/k8s-staging-ci-images/kube-controller-manager@sha256:4660397406f51cc97599ef65f46644d7cffad440bc3169fdfc0224626b3b2780 gcr.io/k8s-staging-ci-images/kube-controller-manager:v1.26.0-alpha.3.425_2f2240400391ad],SizeBytes:31999085,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager@sha256:d3a06262256f3e7578d5f77df137a8cdf58f9f498f35b5b56d116e8a7e31dc91 registry.k8s.io/kube-controller-manager:v1.25.3],SizeBytes:31261869,},ContainerImage{Names:[k8s.gcr.io/metrics-server/metrics-server@sha256:6385aec64bb97040a5e692947107b81e178555c7a5b71caa90d733e4130efc10 k8s.gcr.io/metrics-server/metrics-server:v0.5.2],SizeBytes:26023008,},ContainerImage{Names:[gcr.io/k8s-staging-ci-images/kube-proxy@sha256:8dcde74eedeada05ac0d0b3fec95ca4299544d64c362e7968f7f5ecb69186030 gcr.io/k8s-staging-ci-images/kube-proxy:v1.26.0-alpha.3.425_2f2240400391ad],SizeBytes:21362412,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:6bf25f038543e1f433cb7f2bdda445ed348c7b9279935ebc2ae4f432308ed82f registry.k8s.io/kube-proxy:v1.25.3],SizeBytes:20265805,},ContainerImage{Names:[gcr.io/k8s-staging-ci-images/kube-scheduler@sha256:fb955dc0dd068e8c525ac60042dac1c9cd349272855639394d761125851f8397 gcr.io/k8s-staging-ci-images/kube-scheduler:v1.26.0-alpha.3.425_2f2240400391ad],SizeBytes:17278982,},ContainerImage{Names:[registry.k8s.io/kube-scheduler@sha256:f478aa916568b00269068ff1e9ff742ecc16192eb6e371e30f69f75df904162e registry.k8s.io/kube-scheduler:v1.25.3],SizeBytes:15798744,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[docker.io/sigwindowstools/k8s-gmsa-webhook@sha256:56bba3b5a4788c05dc45f51444b924f6bed05a29789e04e43a6e2c3dbb07d9ce docker.io/sigwindowstools/k8s-gmsa-webhook:latest],SizeBytes:8874257,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Nov  9 03:02:24.568: INFO: 
Logging kubelet events for node capz-conf-tvyqls-control-plane-7vbm8
Nov  9 03:02:24.601: INFO: 
Logging pods the kubelet thinks is on node capz-conf-tvyqls-control-plane-7vbm8
Nov  9 03:02:24.658: INFO: kube-proxy-9ns2c started at 2022-11-09 01:04:42 +0000 UTC (0+1 container statuses recorded)
Nov  9 03:02:24.658: INFO: 	Container kube-proxy ready: true, restart count 0
Nov  9 03:02:24.658: INFO: calico-kube-controllers-56c5ff4bf8-gbkvc started at 2022-11-09 01:05:22 +0000 UTC (0+1 container statuses recorded)
Nov  9 03:02:24.658: INFO: 	Container calico-kube-controllers ready: true, restart count 0
Nov  9 03:02:24.658: INFO: metrics-server-954b56d74-7hmq4 started at 2022-11-09 01:05:30 +0000 UTC (0+1 container statuses recorded)
Nov  9 03:02:24.658: INFO: 	Container metrics-server ready: true, restart count 0
Nov  9 03:02:24.658: INFO: etcd-capz-conf-tvyqls-control-plane-7vbm8 started at 2022-11-09 01:04:42 +0000 UTC (0+1 container statuses recorded)
Nov  9 03:02:24.658: INFO: 	Container etcd ready: true, restart count 0
Nov  9 03:02:24.658: INFO: kube-apiserver-capz-conf-tvyqls-control-plane-7vbm8 started at 2022-11-09 01:04:42 +0000 UTC (0+1 container statuses recorded)
Nov  9 03:02:24.658: INFO: 	Container kube-apiserver ready: true, restart count 0
Nov  9 03:02:24.658: INFO: kube-controller-manager-capz-conf-tvyqls-control-plane-7vbm8 started at 2022-11-09 01:04:42 +0000 UTC (0+1 container statuses recorded)
Nov  9 03:02:24.658: INFO: 	Container kube-controller-manager ready: true, restart count 0
Nov  9 03:02:24.658: INFO: kube-scheduler-capz-conf-tvyqls-control-plane-7vbm8 started at 2022-11-09 01:04:42 +0000 UTC (0+1 container statuses recorded)
Nov  9 03:02:24.658: INFO: 	Container kube-scheduler ready: true, restart count 0
Nov  9 03:02:24.658: INFO: coredns-fdff55fb9-4wwnr started at 2022-11-09 01:10:12 +0000 UTC (0+1 container statuses recorded)
Nov  9 03:02:24.658: INFO: 	Container coredns ready: true, restart count 0
Nov  9 03:02:24.658: INFO: calico-node-7bphx started at 2022-11-09 01:05:02 +0000 UTC (2+1 container statuses recorded)
Nov  9 03:02:24.658: INFO: 	Init container upgrade-ipam ready: true, restart count 0
Nov  9 03:02:24.658: INFO: 	Init container install-cni ready: true, restart count 0
Nov  9 03:02:24.658: INFO: 	Container calico-node ready: true, restart count 0
Nov  9 03:02:24.658: INFO: coredns-fdff55fb9-vl42q started at 2022-11-09 01:10:12 +0000 UTC (0+1 container statuses recorded)
Nov  9 03:02:24.658: INFO: 	Container coredns ready: true, restart count 0
Nov  9 03:02:24.825: INFO: 
Latency metrics for node capz-conf-tvyqls-control-plane-7vbm8
[DeferCleanup (Each)] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: CPU)
  tear down framework | framework.go:193
STEP: Destroying namespace "horizontal-pod-autoscaling-9304" for this suite. 11/09/22 03:02:24.825

				Click to see stdout/stderrfrom junit_01.xml

Filter through log files | View test history on testgrid

Kubernetes e2e suite [It] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: CPU) [Serial] [Slow] Deployment (Pod Resource) Should scale from 1 pod to 3 pods and then from 3 pods to 5 pods using Average Value for aggregation 15m42s

go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\s\[sig\-autoscaling\]\s\[Feature\:HPA\]\sHorizontal\spod\sautoscaling\s\(scale\sresource\:\sCPU\)\s\[Serial\]\s\[Slow\]\sDeployment\s\(Pod\sResource\)\sShould\sscale\sfrom\s1\spod\sto\s3\spods\sand\sthen\sfrom\s3\spods\sto\s5\spods\susing\sAverage\sValue\sfor\saggregation$'

test/e2e/autoscaling/horizontal_pod_autoscaling.go:209
k8s.io/kubernetes/test/e2e/autoscaling.(*HPAScaleTest).run(0xc0031dbe68, {0x74e3c28?, 0xc0015a0c00?}, {{0x74b92e8, 0x4}, {0x74c2479, 0x7}, {0x74ca886, 0xa}}, 0xc00082cd20)
	test/e2e/autoscaling/horizontal_pod_autoscaling.go:209 +0x2d8
k8s.io/kubernetes/test/e2e/autoscaling.scaleUp({0x74e3c28?, 0x61dc905?}, {{0x74b92e8, 0x4}, {0x74c2479, 0x7}, {0x74ca886, 0xa}}, {0x74b8538, 0x3}, ...)
	test/e2e/autoscaling/horizontal_pod_autoscaling.go:249 +0x212
k8s.io/kubernetes/test/e2e/autoscaling.glob..func6.1.3()
	test/e2e/autoscaling/horizontal_pod_autoscaling.go:56 +0x88
				[BeforeEach] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: CPU)
  set up framework | framework.go:178
STEP: Creating a kubernetes client 11/09/22 04:34:41.746
Nov  9 04:34:41.746: INFO: >>> kubeConfig: /home/prow/go/src/k8s.io/windows-testing/capz-conf-tvyqls.kubeconfig
I1109 04:34:41.747851   14108 discovery.go:214] Invalidating discovery information
STEP: Building a namespace api object, basename horizontal-pod-autoscaling 11/09/22 04:34:41.747
I1109 04:34:41.785189   14108 reflector.go:221] Starting reflector *v1.ServiceAccount (0s) from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
I1109 04:34:41.785226   14108 reflector.go:257] Listing and watching *v1.ServiceAccount from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
I1109 04:34:41.847122   14108 reflector.go:227] Stopping reflector *v1.ServiceAccount (0s) from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
STEP: Waiting for a default service account to be provisioned in namespace 11/09/22 04:34:41.847
I1109 04:34:41.847396   14108 reflector.go:221] Starting reflector *v1.ServiceAccount (0s) from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
I1109 04:34:41.847493   14108 reflector.go:257] Listing and watching *v1.ServiceAccount from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
I1109 04:34:41.909505   14108 reflector.go:227] Stopping reflector *v1.ServiceAccount (0s) from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
STEP: Waiting for kube-root-ca.crt to be provisioned in namespace 11/09/22 04:34:41.909
I1109 04:34:41.909754   14108 reflector.go:221] Starting reflector *v1.ConfigMap (0s) from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
I1109 04:34:41.909771   14108 reflector.go:257] Listing and watching *v1.ConfigMap from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
I1109 04:34:41.971662   14108 reflector.go:227] Stopping reflector *v1.ConfigMap (0s) from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
[BeforeEach] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: CPU)
  test/e2e/framework/metrics/init/init.go:31
[It] Should scale from 1 pod to 3 pods and then from 3 pods to 5 pods using Average Value for aggregation
  test/e2e/autoscaling/horizontal_pod_autoscaling.go:55
Nov  9 04:34:41.971: INFO: >>> kubeConfig: /home/prow/go/src/k8s.io/windows-testing/capz-conf-tvyqls.kubeconfig
STEP: Running consuming RC test-deployment via apps/v1beta2, Kind=Deployment with 1 replicas 11/09/22 04:34:41.972
STEP: Creating deployment test-deployment in namespace horizontal-pod-autoscaling-2763 11/09/22 04:34:42.016
I1109 04:34:42.055546   14108 runners.go:193] Created deployment with name: test-deployment, namespace: horizontal-pod-autoscaling-2763, replica count: 1
I1109 04:34:42.055660   14108 reflector.go:221] Starting reflector *v1.Pod (0s) from test/utils/pod_store.go:57
I1109 04:34:42.055673   14108 reflector.go:257] Listing and watching *v1.Pod from test/utils/pod_store.go:57
I1109 04:34:52.107166   14108 runners.go:193] test-deployment Pods: 1 out of 1 created, 1 running, 0 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1109 04:34:52.107567   14108 reflector.go:227] Stopping reflector *v1.Pod (0s) from test/utils/pod_store.go:57
STEP: Running controller 11/09/22 04:34:52.107
STEP: creating replication controller test-deployment-ctrl in namespace horizontal-pod-autoscaling-2763 11/09/22 04:34:52.154
I1109 04:34:52.193174   14108 runners.go:193] Created replication controller with name: test-deployment-ctrl, namespace: horizontal-pod-autoscaling-2763, replica count: 1
I1109 04:34:52.193329   14108 reflector.go:221] Starting reflector *v1.Pod (0s) from test/utils/pod_store.go:57
I1109 04:34:52.193339   14108 reflector.go:257] Listing and watching *v1.Pod from test/utils/pod_store.go:57
I1109 04:35:02.244507   14108 runners.go:193] test-deployment-ctrl Pods: 1 out of 1 created, 1 running, 0 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1109 04:35:02.244817   14108 reflector.go:227] Stopping reflector *v1.Pod (0s) from test/utils/pod_store.go:57
Nov  9 04:35:07.245: INFO: Waiting for amount of service:test-deployment-ctrl endpoints to be 1
Nov  9 04:35:07.277: INFO: RC test-deployment: consume 250 millicores in total
Nov  9 04:35:07.278: INFO: RC test-deployment: consume 0 MB in total
Nov  9 04:35:07.278: INFO: RC test-deployment: setting consumption to 250 millicores in total
Nov  9 04:35:07.278: INFO: RC test-deployment: consume custom metric 0 in total
Nov  9 04:35:07.278: INFO: RC test-deployment: disabling mem consumption
Nov  9 04:35:07.278: INFO: RC test-deployment: disabling consumption of custom metric QPS
Nov  9 04:35:07.347: INFO: waiting for 3 replicas (current: 1)
Nov  9 04:35:27.380: INFO: waiting for 3 replicas (current: 1)
Nov  9 04:35:37.278: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 04:35:37.278: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-2763/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 04:35:47.380: INFO: waiting for 3 replicas (current: 1)
Nov  9 04:36:07.356: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 04:36:07.356: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-2763/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 04:36:07.380: INFO: waiting for 3 replicas (current: 1)
Nov  9 04:36:27.382: INFO: waiting for 3 replicas (current: 2)
Nov  9 04:36:37.406: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 04:36:37.406: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-2763/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 04:36:47.382: INFO: waiting for 3 replicas (current: 2)
Nov  9 04:37:07.381: INFO: waiting for 3 replicas (current: 2)
Nov  9 04:37:07.454: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 04:37:07.454: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-2763/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 04:37:27.380: INFO: waiting for 3 replicas (current: 2)
Nov  9 04:37:37.503: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 04:37:37.503: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-2763/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 04:37:47.381: INFO: waiting for 3 replicas (current: 2)
Nov  9 04:38:07.381: INFO: waiting for 3 replicas (current: 2)
Nov  9 04:38:07.551: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 04:38:07.551: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-2763/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 04:38:27.380: INFO: waiting for 3 replicas (current: 2)
Nov  9 04:38:37.596: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 04:38:37.596: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-2763/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 04:38:47.380: INFO: waiting for 3 replicas (current: 2)
Nov  9 04:39:07.381: INFO: waiting for 3 replicas (current: 2)
Nov  9 04:39:07.644: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 04:39:07.644: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-2763/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 04:39:27.381: INFO: waiting for 3 replicas (current: 2)
Nov  9 04:39:37.690: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 04:39:37.690: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-2763/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 04:39:47.381: INFO: waiting for 3 replicas (current: 2)
Nov  9 04:40:07.382: INFO: waiting for 3 replicas (current: 2)
Nov  9 04:40:07.736: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 04:40:07.736: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-2763/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 04:40:27.381: INFO: waiting for 3 replicas (current: 2)
Nov  9 04:40:37.777: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 04:40:37.777: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-2763/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 04:40:47.380: INFO: waiting for 3 replicas (current: 2)
Nov  9 04:41:07.382: INFO: waiting for 3 replicas (current: 2)
Nov  9 04:41:07.822: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 04:41:07.822: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-2763/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 04:41:27.382: INFO: waiting for 3 replicas (current: 2)
Nov  9 04:41:37.868: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 04:41:37.869: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-2763/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 04:41:47.381: INFO: waiting for 3 replicas (current: 2)
Nov  9 04:42:07.382: INFO: waiting for 3 replicas (current: 2)
Nov  9 04:42:07.919: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 04:42:07.919: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-2763/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 04:42:27.381: INFO: waiting for 3 replicas (current: 2)
Nov  9 04:42:37.967: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 04:42:37.967: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-2763/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 04:42:47.382: INFO: waiting for 3 replicas (current: 2)
Nov  9 04:43:07.381: INFO: waiting for 3 replicas (current: 2)
Nov  9 04:43:08.013: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 04:43:08.013: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-2763/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 04:43:27.382: INFO: waiting for 3 replicas (current: 2)
Nov  9 04:43:38.058: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 04:43:38.058: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-2763/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 04:43:47.382: INFO: waiting for 3 replicas (current: 2)
Nov  9 04:44:07.382: INFO: waiting for 3 replicas (current: 2)
Nov  9 04:44:08.106: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 04:44:08.106: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-2763/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 04:44:27.381: INFO: waiting for 3 replicas (current: 2)
Nov  9 04:44:38.154: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 04:44:38.154: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-2763/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 04:44:47.382: INFO: waiting for 3 replicas (current: 2)
Nov  9 04:45:07.381: INFO: waiting for 3 replicas (current: 2)
Nov  9 04:45:08.200: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 04:45:08.200: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-2763/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 04:45:27.381: INFO: waiting for 3 replicas (current: 2)
Nov  9 04:45:38.242: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 04:45:38.242: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-2763/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 04:45:47.381: INFO: waiting for 3 replicas (current: 2)
Nov  9 04:46:07.381: INFO: waiting for 3 replicas (current: 2)
Nov  9 04:46:08.291: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 04:46:08.291: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-2763/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 04:46:27.381: INFO: waiting for 3 replicas (current: 2)
Nov  9 04:46:38.338: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 04:46:38.339: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-2763/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 04:46:47.381: INFO: waiting for 3 replicas (current: 2)
Nov  9 04:47:07.382: INFO: waiting for 3 replicas (current: 2)
Nov  9 04:47:08.385: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 04:47:08.385: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-2763/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 04:47:27.382: INFO: waiting for 3 replicas (current: 2)
Nov  9 04:47:38.433: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 04:47:38.433: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-2763/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 04:47:47.381: INFO: waiting for 3 replicas (current: 2)
Nov  9 04:48:07.381: INFO: waiting for 3 replicas (current: 2)
Nov  9 04:48:08.481: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 04:48:08.481: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-2763/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 04:48:27.381: INFO: waiting for 3 replicas (current: 2)
Nov  9 04:48:38.526: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 04:48:38.526: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-2763/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 04:48:47.382: INFO: waiting for 3 replicas (current: 2)
Nov  9 04:49:07.381: INFO: waiting for 3 replicas (current: 2)
Nov  9 04:49:08.571: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 04:49:08.571: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-2763/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 04:49:27.381: INFO: waiting for 3 replicas (current: 2)
Nov  9 04:49:38.616: INFO: RC test-deployment: sending request to consume 250 millicores
Nov  9 04:49:38.616: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-2763/services/test-deployment-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 04:49:47.384: INFO: waiting for 3 replicas (current: 2)
Nov  9 04:50:07.380: INFO: waiting for 3 replicas (current: 2)
Nov  9 04:50:07.412: INFO: waiting for 3 replicas (current: 2)
Nov  9 04:50:07.412: INFO: Unexpected error: timeout waiting 15m0s for 3 replicas: 
    <*errors.errorString | 0xc000287c80>: {
        s: "timed out waiting for the condition",
    }
Nov  9 04:50:07.412: FAIL: timeout waiting 15m0s for 3 replicas: timed out waiting for the condition

Full Stack Trace
k8s.io/kubernetes/test/e2e/autoscaling.(*HPAScaleTest).run(0xc0031dbe68, {0x74e3c28?, 0xc0015a0c00?}, {{0x74b92e8, 0x4}, {0x74c2479, 0x7}, {0x74ca886, 0xa}}, 0xc00082cd20)
	test/e2e/autoscaling/horizontal_pod_autoscaling.go:209 +0x2d8
k8s.io/kubernetes/test/e2e/autoscaling.scaleUp({0x74e3c28?, 0x61dc905?}, {{0x74b92e8, 0x4}, {0x74c2479, 0x7}, {0x74ca886, 0xa}}, {0x74b8538, 0x3}, ...)
	test/e2e/autoscaling/horizontal_pod_autoscaling.go:249 +0x212
k8s.io/kubernetes/test/e2e/autoscaling.glob..func6.1.3()
	test/e2e/autoscaling/horizontal_pod_autoscaling.go:56 +0x88
STEP: Removing consuming RC test-deployment 11/09/22 04:50:07.448
Nov  9 04:50:07.448: INFO: RC test-deployment: stopping metric consumer
Nov  9 04:50:07.448: INFO: RC test-deployment: stopping CPU consumer
Nov  9 04:50:07.448: INFO: RC test-deployment: stopping mem consumer
STEP: deleting Deployment.apps test-deployment in namespace horizontal-pod-autoscaling-2763, will wait for the garbage collector to delete the pods 11/09/22 04:50:17.448
I1109 04:50:17.488930   14108 reflector.go:221] Starting reflector *v1.Pod (0s) from test/utils/pod_store.go:57
I1109 04:50:17.489000   14108 reflector.go:257] Listing and watching *v1.Pod from test/utils/pod_store.go:57
Nov  9 04:50:17.575: INFO: Deleting Deployment.apps test-deployment took: 35.811949ms
I1109 04:50:17.676255   14108 resource.go:655] Ignoring inactive pod horizontal-pod-autoscaling-2763/test-deployment-669bb6996d-7tkps in state Running, deletion time 2022-11-09 04:50:47 +0000 UTC
I1109 04:50:17.676320   14108 resource.go:655] Ignoring inactive pod horizontal-pod-autoscaling-2763/test-deployment-669bb6996d-djbhw in state Running, deletion time 2022-11-09 04:50:47 +0000 UTC
Nov  9 04:50:17.676: INFO: Terminating Deployment.apps test-deployment pods took: 100.939899ms
I1109 04:50:19.977157   14108 reflector.go:227] Stopping reflector *v1.Pod (0s) from test/utils/pod_store.go:57
STEP: deleting ReplicationController test-deployment-ctrl in namespace horizontal-pod-autoscaling-2763, will wait for the garbage collector to delete the pods 11/09/22 04:50:20.039
I1109 04:50:20.074800   14108 reflector.go:221] Starting reflector *v1.Pod (0s) from test/utils/pod_store.go:57
I1109 04:50:20.074832   14108 reflector.go:257] Listing and watching *v1.Pod from test/utils/pod_store.go:57
Nov  9 04:50:20.166: INFO: Deleting ReplicationController test-deployment-ctrl took: 40.659418ms
I1109 04:50:20.267384   14108 resource.go:655] Ignoring inactive pod horizontal-pod-autoscaling-2763/test-deployment-ctrl-vtzss in state Running, deletion time 2022-11-09 04:50:21 +0000 UTC
Nov  9 04:50:20.267: INFO: Terminating ReplicationController test-deployment-ctrl pods took: 101.016868ms
I1109 04:50:22.768487   14108 reflector.go:227] Stopping reflector *v1.Pod (0s) from test/utils/pod_store.go:57
[AfterEach] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: CPU)
  test/e2e/framework/node/init/init.go:32
Nov  9 04:50:22.836: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
[DeferCleanup (Each)] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: CPU)
  test/e2e/framework/metrics/init/init.go:33
[DeferCleanup (Each)] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: CPU)
  dump namespaces | framework.go:196
STEP: dump namespace information after failure 11/09/22 04:50:22.87
STEP: Collecting events from namespace "horizontal-pod-autoscaling-2763". 11/09/22 04:50:22.87
STEP: Found 21 events. 11/09/22 04:50:22.904
Nov  9 04:50:22.904: INFO: At 2022-11-09 04:34:42 +0000 UTC - event for test-deployment: {deployment-controller } ScalingReplicaSet: Scaled up replica set test-deployment-669bb6996d to 1
Nov  9 04:50:22.904: INFO: At 2022-11-09 04:34:42 +0000 UTC - event for test-deployment-669bb6996d: {replicaset-controller } SuccessfulCreate: Created pod: test-deployment-669bb6996d-7tkps
Nov  9 04:50:22.904: INFO: At 2022-11-09 04:34:42 +0000 UTC - event for test-deployment-669bb6996d-7tkps: {default-scheduler } Scheduled: Successfully assigned horizontal-pod-autoscaling-2763/test-deployment-669bb6996d-7tkps to capz-conf-hhzhh
Nov  9 04:50:22.904: INFO: At 2022-11-09 04:34:45 +0000 UTC - event for test-deployment-669bb6996d-7tkps: {kubelet capz-conf-hhzhh} Pulled: Container image "registry.k8s.io/e2e-test-images/resource-consumer:1.13" already present on machine
Nov  9 04:50:22.904: INFO: At 2022-11-09 04:34:45 +0000 UTC - event for test-deployment-669bb6996d-7tkps: {kubelet capz-conf-hhzhh} Created: Created container test-deployment
Nov  9 04:50:22.904: INFO: At 2022-11-09 04:34:47 +0000 UTC - event for test-deployment-669bb6996d-7tkps: {kubelet capz-conf-hhzhh} Started: Started container test-deployment
Nov  9 04:50:22.904: INFO: At 2022-11-09 04:34:52 +0000 UTC - event for test-deployment-ctrl: {replication-controller } SuccessfulCreate: Created pod: test-deployment-ctrl-vtzss
Nov  9 04:50:22.904: INFO: At 2022-11-09 04:34:52 +0000 UTC - event for test-deployment-ctrl-vtzss: {default-scheduler } Scheduled: Successfully assigned horizontal-pod-autoscaling-2763/test-deployment-ctrl-vtzss to capz-conf-m97cx
Nov  9 04:50:22.904: INFO: At 2022-11-09 04:34:54 +0000 UTC - event for test-deployment-ctrl-vtzss: {kubelet capz-conf-m97cx} Created: Created container test-deployment-ctrl
Nov  9 04:50:22.904: INFO: At 2022-11-09 04:34:54 +0000 UTC - event for test-deployment-ctrl-vtzss: {kubelet capz-conf-m97cx} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.40" already present on machine
Nov  9 04:50:22.904: INFO: At 2022-11-09 04:34:55 +0000 UTC - event for test-deployment-ctrl-vtzss: {kubelet capz-conf-m97cx} Started: Started container test-deployment-ctrl
Nov  9 04:50:22.904: INFO: At 2022-11-09 04:36:07 +0000 UTC - event for test-deployment: {horizontal-pod-autoscaler } SuccessfulRescale: New size: 2; reason: cpu resource above target
Nov  9 04:50:22.904: INFO: At 2022-11-09 04:36:07 +0000 UTC - event for test-deployment: {deployment-controller } ScalingReplicaSet: Scaled up replica set test-deployment-669bb6996d to 2 from 1
Nov  9 04:50:22.904: INFO: At 2022-11-09 04:36:07 +0000 UTC - event for test-deployment-669bb6996d: {replicaset-controller } SuccessfulCreate: Created pod: test-deployment-669bb6996d-djbhw
Nov  9 04:50:22.904: INFO: At 2022-11-09 04:36:07 +0000 UTC - event for test-deployment-669bb6996d-djbhw: {default-scheduler } Scheduled: Successfully assigned horizontal-pod-autoscaling-2763/test-deployment-669bb6996d-djbhw to capz-conf-m97cx
Nov  9 04:50:22.904: INFO: At 2022-11-09 04:36:09 +0000 UTC - event for test-deployment-669bb6996d-djbhw: {kubelet capz-conf-m97cx} Created: Created container test-deployment
Nov  9 04:50:22.904: INFO: At 2022-11-09 04:36:09 +0000 UTC - event for test-deployment-669bb6996d-djbhw: {kubelet capz-conf-m97cx} Pulled: Container image "registry.k8s.io/e2e-test-images/resource-consumer:1.13" already present on machine
Nov  9 04:50:22.904: INFO: At 2022-11-09 04:36:11 +0000 UTC - event for test-deployment-669bb6996d-djbhw: {kubelet capz-conf-m97cx} Started: Started container test-deployment
Nov  9 04:50:22.904: INFO: At 2022-11-09 04:50:17 +0000 UTC - event for test-deployment-669bb6996d-7tkps: {kubelet capz-conf-hhzhh} Killing: Stopping container test-deployment
Nov  9 04:50:22.904: INFO: At 2022-11-09 04:50:17 +0000 UTC - event for test-deployment-669bb6996d-djbhw: {kubelet capz-conf-m97cx} Killing: Stopping container test-deployment
Nov  9 04:50:22.904: INFO: At 2022-11-09 04:50:20 +0000 UTC - event for test-deployment-ctrl-vtzss: {kubelet capz-conf-m97cx} Killing: Stopping container test-deployment-ctrl
Nov  9 04:50:22.939: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
Nov  9 04:50:22.939: INFO: 
Nov  9 04:50:22.974: INFO: 
Logging node info for node capz-conf-hhzhh
Nov  9 04:50:23.012: INFO: Node Info: &Node{ObjectMeta:{capz-conf-hhzhh    641b6838-eeea-48b0-9dc0-0ff9ab096f10 32223 0 2022-11-09 01:07:16 +0000 UTC <nil> <nil> map[agentpool:windowsgmsa beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:windows failure-domain.beta.kubernetes.io/region:eastus failure-domain.beta.kubernetes.io/zone:0 kubernetes.io/arch:amd64 kubernetes.io/hostname:capz-conf-hhzhh kubernetes.io/os:windows node.kubernetes.io/instance-type:Standard_D4s_v3 node.kubernetes.io/windows-build:10.0.20348 topology.kubernetes.io/region:eastus topology.kubernetes.io/zone:0] map[cluster.x-k8s.io/cluster-name:capz-conf-tvyqls cluster.x-k8s.io/cluster-namespace:default cluster.x-k8s.io/machine:capz-conf-tvyqls-md-win-7848cfdb84-9mln2 cluster.x-k8s.io/owner-kind:MachineSet cluster.x-k8s.io/owner-name:capz-conf-tvyqls-md-win-7848cfdb84 kubeadm.alpha.kubernetes.io/cri-socket:npipe:////./pipe/containerd-containerd node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:10.1.0.4/16 projectcalico.org/IPv4VXLANTunnelAddr:192.168.84.65 projectcalico.org/VXLANTunnelMACAddr:00:15:5d:5b:b6:ce volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubeadm Update v1 2022-11-09 01:07:16 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:kubeadm.alpha.kubernetes.io/cri-socket":{}}}} } {kubelet.exe Update v1 2022-11-09 01:07:16 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:node.kubernetes.io/windows-build":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {manager Update v1 2022-11-09 01:08:02 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:cluster.x-k8s.io/cluster-name":{},"f:cluster.x-k8s.io/cluster-namespace":{},"f:cluster.x-k8s.io/machine":{},"f:cluster.x-k8s.io/owner-kind":{},"f:cluster.x-k8s.io/owner-name":{}}}} } {kube-controller-manager Update v1 2022-11-09 01:08:06 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}}} } {Go-http-client Update v1 2022-11-09 01:08:32 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4VXLANTunnelAddr":{},"f:projectcalico.org/VXLANTunnelMACAddr":{}}}} status} {cluster-api-e2e Update v1 2022-11-09 01:10:08 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:agentpool":{}}}} } {e2e.test Update v1 2022-11-09 03:54:19 +0000 UTC FieldsV1 {"f:status":{"f:capacity":{"f:example.com/fakePTSRes":{}}}} status} {kubelet.exe Update v1 2022-11-09 04:47:17 +0000 UTC FieldsV1 {"f:status":{"f:allocatable":{"f:example.com/fakePTSRes":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:,DoNotUseExternalID:,ProviderID:azure:///subscriptions/===REDACTED===/resourceGroups/capz-conf-tvyqls/providers/Microsoft.Compute/virtualMachines/capz-conf-hhzhh,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-azure-disk: {{8 0} {<nil>} 8 DecimalSI},cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{136912564224 0} {<nil>} 133703676Ki BinarySI},example.com/fakePTSRes: {{10 0} {<nil>} 10 DecimalSI},memory: {{17179398144 0} {<nil>} 16776756Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-azure-disk: {{8 0} {<nil>} 8 DecimalSI},cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{123221307598 0} {<nil>} 123221307598 DecimalSI},example.com/fakePTSRes: {{10 0} {<nil>} 10 DecimalSI},memory: {{17074540544 0} {<nil>} 16674356Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-09 04:47:17 +0000 UTC,LastTransitionTime:2022-11-09 01:07:16 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-09 04:47:17 +0000 UTC,LastTransitionTime:2022-11-09 01:07:16 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-09 04:47:17 +0000 UTC,LastTransitionTime:2022-11-09 01:07:16 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-09 04:47:17 +0000 UTC,LastTransitionTime:2022-11-09 01:08:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status,},},Addresses:[]NodeAddress{NodeAddress{Type:Hostname,Address:capz-conf-hhzhh,},NodeAddress{Type:InternalIP,Address:10.1.0.4,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:capz-conf-hhzhh,SystemUUID:5F9EA3E7-22F1-406E-9AAC-69BA5DD83BF4,BootID:8,KernelVersion:10.0.20348.1129,OSImage:Windows Server 2022 Datacenter,ContainerRuntimeVersion:containerd://1.6.8,KubeletVersion:v1.26.0-alpha.3.425+2f2240400391ad,KubeProxyVersion:v1.26.0-alpha.3.425+2f2240400391ad,OperatingSystem:windows,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:307641217,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:221349084,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:218942479,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:183620591,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:181784786,},ContainerImage{Names:[ghcr.io/kubernetes-sigs/sig-windows/eventflow-logger@sha256:63bf2aa9db909d0d90fb5205abf7fb2a6d9a494b89cbd2508a42457dfc875505 ghcr.io/kubernetes-sigs/sig-windows/eventflow-logger:v0.1.0],SizeBytes:148054279,},ContainerImage{Names:[ghcr.io/kubernetes-sigs/sig-windows/csi-proxy@sha256:96b4144986319a747ba599892454be2737aae6005d96b8e13ed481321ac3afba ghcr.io/kubernetes-sigs/sig-windows/csi-proxy:v1.0.2],SizeBytes:123925937,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/resource-consumer@sha256:ba5e047a337e5d0709bc57df45b95b2c7f6f2794b290e4e24f7fc8980d60b25a registry.k8s.io/e2e-test-images/resource-consumer:1.13],SizeBytes:121182992,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6],SizeBytes:119397646,},ContainerImage{Names:[docker.io/sigwindowstools/kube-proxy:v1.23.1-calico-hostprocess docker.io/sigwindowstools/kube-proxy:v1.26.0-alpha.3.425_2f2240400391ad-calico-hostprocess],SizeBytes:116182072,},ContainerImage{Names:[docker.io/sigwindowstools/calico-install@sha256:97bc10aa5000a0ee1c842ac32771fe7a45a3a5ca507711bdf57ae2eb5f293e2b docker.io/sigwindowstools/calico-install:v3.23.0-hostprocess],SizeBytes:47258343,},ContainerImage{Names:[docker.io/sigwindowstools/calico-node@sha256:ea8b55bde9aed6a649582a6e21029577430661c743d94b3a5e93d57e648874a2 docker.io/sigwindowstools/calico-node:v3.23.0-hostprocess],SizeBytes:27005624,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Nov  9 04:50:23.012: INFO: 
Logging kubelet events for node capz-conf-hhzhh
Nov  9 04:50:23.045: INFO: 
Logging pods the kubelet thinks is on node capz-conf-hhzhh
Nov  9 04:50:23.098: INFO: calico-node-windows-jpm48 started at 2022-11-09 01:07:16 +0000 UTC (1+2 container statuses recorded)
Nov  9 04:50:23.098: INFO: 	Init container install-cni ready: true, restart count 0
Nov  9 04:50:23.098: INFO: 	Container calico-node-felix ready: true, restart count 1
Nov  9 04:50:23.098: INFO: 	Container calico-node-startup ready: true, restart count 0
Nov  9 04:50:23.098: INFO: csi-proxy-dkn8s started at 2022-11-09 01:08:06 +0000 UTC (0+1 container statuses recorded)
Nov  9 04:50:23.098: INFO: 	Container csi-proxy ready: true, restart count 0
Nov  9 04:50:23.098: INFO: containerd-logger-z56l7 started at 2022-11-09 01:07:16 +0000 UTC (0+1 container statuses recorded)
Nov  9 04:50:23.098: INFO: 	Container containerd-logger ready: true, restart count 0
Nov  9 04:50:23.098: INFO: kube-proxy-windows-ldnh5 started at 2022-11-09 01:07:16 +0000 UTC (0+1 container statuses recorded)
Nov  9 04:50:23.098: INFO: 	Container kube-proxy ready: true, restart count 0
Nov  9 04:50:23.288: INFO: 
Latency metrics for node capz-conf-hhzhh
Nov  9 04:50:23.288: INFO: 
Logging node info for node capz-conf-m97cx
Nov  9 04:50:23.321: INFO: Node Info: &Node{ObjectMeta:{capz-conf-m97cx    36965107-96dc-426b-924d-c291c94a2d04 32229 0 2022-11-09 01:07:00 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:windows failure-domain.beta.kubernetes.io/region:eastus failure-domain.beta.kubernetes.io/zone:0 kubernetes.io/arch:amd64 kubernetes.io/hostname:capz-conf-m97cx kubernetes.io/os:windows node.kubernetes.io/instance-type:Standard_D4s_v3 node.kubernetes.io/windows-build:10.0.20348 topology.kubernetes.io/region:eastus topology.kubernetes.io/zone:0] map[cluster.x-k8s.io/cluster-name:capz-conf-tvyqls cluster.x-k8s.io/cluster-namespace:default cluster.x-k8s.io/machine:capz-conf-tvyqls-md-win-7848cfdb84-cw8bk cluster.x-k8s.io/owner-kind:MachineSet cluster.x-k8s.io/owner-name:capz-conf-tvyqls-md-win-7848cfdb84 kubeadm.alpha.kubernetes.io/cri-socket:npipe:////./pipe/containerd-containerd node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:10.1.0.5/16 projectcalico.org/IPv4VXLANTunnelAddr:192.168.100.1 projectcalico.org/VXLANTunnelMACAddr:00:15:5d:ac:92:68 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubeadm Update v1 2022-11-09 01:07:00 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:kubeadm.alpha.kubernetes.io/cri-socket":{}}}} } {kubelet.exe Update v1 2022-11-09 01:07:00 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:node.kubernetes.io/windows-build":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-11-09 01:07:31 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}}} } {manager Update v1 2022-11-09 01:08:02 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:cluster.x-k8s.io/cluster-name":{},"f:cluster.x-k8s.io/cluster-namespace":{},"f:cluster.x-k8s.io/machine":{},"f:cluster.x-k8s.io/owner-kind":{},"f:cluster.x-k8s.io/owner-name":{}}}} } {Go-http-client Update v1 2022-11-09 01:08:08 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4VXLANTunnelAddr":{},"f:projectcalico.org/VXLANTunnelMACAddr":{}}}} status} {e2e.test Update v1 2022-11-09 03:54:19 +0000 UTC FieldsV1 {"f:status":{"f:capacity":{"f:example.com/fakePTSRes":{},"f:example.com/fakecpu":{}}}} status} {kubelet.exe Update v1 2022-11-09 04:47:20 +0000 UTC FieldsV1 {"f:status":{"f:allocatable":{"f:example.com/fakePTSRes":{},"f:example.com/fakecpu":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:,DoNotUseExternalID:,ProviderID:azure:///subscriptions/===REDACTED===/resourceGroups/capz-conf-tvyqls/providers/Microsoft.Compute/virtualMachines/capz-conf-m97cx,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-azure-disk: {{8 0} {<nil>} 8 DecimalSI},cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{136912564224 0} {<nil>} 133703676Ki BinarySI},example.com/fakePTSRes: {{10 0} {<nil>} 10 DecimalSI},example.com/fakecpu: {{1 3} {<nil>} 1k DecimalSI},memory: {{17179398144 0} {<nil>} 16776756Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-azure-disk: {{8 0} {<nil>} 8 DecimalSI},cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{123221307598 0} {<nil>} 123221307598 DecimalSI},example.com/fakePTSRes: {{10 0} {<nil>} 10 DecimalSI},example.com/fakecpu: {{1 3} {<nil>} 1k DecimalSI},memory: {{17074540544 0} {<nil>} 16674356Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-09 04:47:20 +0000 UTC,LastTransitionTime:2022-11-09 01:07:00 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-09 04:47:20 +0000 UTC,LastTransitionTime:2022-11-09 01:07:00 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-09 04:47:20 +0000 UTC,LastTransitionTime:2022-11-09 01:07:00 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-09 04:47:20 +0000 UTC,LastTransitionTime:2022-11-09 01:07:31 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status,},},Addresses:[]NodeAddress{NodeAddress{Type:Hostname,Address:capz-conf-m97cx,},NodeAddress{Type:InternalIP,Address:10.1.0.5,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:capz-conf-m97cx,SystemUUID:FC504DDA-1329-4BBA-8C1E-77FAD783BCA1,BootID:8,KernelVersion:10.0.20348.1129,OSImage:Windows Server 2022 Datacenter,ContainerRuntimeVersion:containerd://1.6.8,KubeletVersion:v1.26.0-alpha.3.425+2f2240400391ad,KubeProxyVersion:v1.26.0-alpha.3.425+2f2240400391ad,OperatingSystem:windows,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:307641217,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:221349084,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:218942479,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:183620591,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:181784786,},ContainerImage{Names:[ghcr.io/kubernetes-sigs/sig-windows/eventflow-logger@sha256:63bf2aa9db909d0d90fb5205abf7fb2a6d9a494b89cbd2508a42457dfc875505 ghcr.io/kubernetes-sigs/sig-windows/eventflow-logger:v0.1.0],SizeBytes:148054279,},ContainerImage{Names:[ghcr.io/kubernetes-sigs/sig-windows/csi-proxy@sha256:96b4144986319a747ba599892454be2737aae6005d96b8e13ed481321ac3afba ghcr.io/kubernetes-sigs/sig-windows/csi-proxy:v1.0.2],SizeBytes:123925937,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/resource-consumer@sha256:ba5e047a337e5d0709bc57df45b95b2c7f6f2794b290e4e24f7fc8980d60b25a registry.k8s.io/e2e-test-images/resource-consumer:1.13],SizeBytes:121182992,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6],SizeBytes:119397646,},ContainerImage{Names:[docker.io/sigwindowstools/kube-proxy:v1.23.1-calico-hostprocess docker.io/sigwindowstools/kube-proxy:v1.26.0-alpha.3.425_2f2240400391ad-calico-hostprocess],SizeBytes:116182072,},ContainerImage{Names:[docker.io/sigwindowstools/calico-install@sha256:97bc10aa5000a0ee1c842ac32771fe7a45a3a5ca507711bdf57ae2eb5f293e2b docker.io/sigwindowstools/calico-install:v3.23.0-hostprocess],SizeBytes:47258343,},ContainerImage{Names:[docker.io/sigwindowstools/calico-node@sha256:ea8b55bde9aed6a649582a6e21029577430661c743d94b3a5e93d57e648874a2 docker.io/sigwindowstools/calico-node:v3.23.0-hostprocess],SizeBytes:27005624,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Nov  9 04:50:23.321: INFO: 
Logging kubelet events for node capz-conf-m97cx
Nov  9 04:50:23.353: INFO: 
Logging pods the kubelet thinks is on node capz-conf-m97cx
Nov  9 04:50:23.404: INFO: kube-proxy-windows-55jdx started at 2022-11-09 01:07:01 +0000 UTC (0+1 container statuses recorded)
Nov  9 04:50:23.404: INFO: 	Container kube-proxy ready: true, restart count 0
Nov  9 04:50:23.404: INFO: csi-proxy-tn9x7 started at 2022-11-09 01:07:31 +0000 UTC (0+1 container statuses recorded)
Nov  9 04:50:23.404: INFO: 	Container csi-proxy ready: true, restart count 0
Nov  9 04:50:23.404: INFO: calico-node-windows-w5q9w started at 2022-11-09 01:07:01 +0000 UTC (1+2 container statuses recorded)
Nov  9 04:50:23.404: INFO: 	Init container install-cni ready: true, restart count 0
Nov  9 04:50:23.404: INFO: 	Container calico-node-felix ready: true, restart count 1
Nov  9 04:50:23.404: INFO: 	Container calico-node-startup ready: true, restart count 0
Nov  9 04:50:23.404: INFO: containerd-logger-v9qh5 started at 2022-11-09 01:07:01 +0000 UTC (0+1 container statuses recorded)
Nov  9 04:50:23.404: INFO: 	Container containerd-logger ready: true, restart count 0
Nov  9 04:50:23.568: INFO: 
Latency metrics for node capz-conf-m97cx
Nov  9 04:50:23.568: INFO: 
Logging node info for node capz-conf-tvyqls-control-plane-7vbm8
Nov  9 04:50:23.601: INFO: Node Info: &Node{ObjectMeta:{capz-conf-tvyqls-control-plane-7vbm8    59067037-dec8-4e79-8a8a-d150ea7ce6ba 32191 0 2022-11-09 01:04:38 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D2s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eastus failure-domain.beta.kubernetes.io/zone:eastus-3 kubernetes.io/arch:amd64 kubernetes.io/hostname:capz-conf-tvyqls-control-plane-7vbm8 kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:Standard_D2s_v3 topology.kubernetes.io/region:eastus topology.kubernetes.io/zone:eastus-3] map[cluster.x-k8s.io/cluster-name:capz-conf-tvyqls cluster.x-k8s.io/cluster-namespace:default cluster.x-k8s.io/machine:capz-conf-tvyqls-control-plane-ljqrh cluster.x-k8s.io/owner-kind:KubeadmControlPlane cluster.x-k8s.io/owner-name:capz-conf-tvyqls-control-plane kubeadm.alpha.kubernetes.io/cri-socket:unix:///var/run/containerd/containerd.sock node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:10.0.0.4/16 projectcalico.org/IPv4VXLANTunnelAddr:192.168.2.192 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-11-09 01:04:38 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kubeadm Update v1 2022-11-09 01:04:40 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:kubeadm.alpha.kubernetes.io/cri-socket":{}},"f:labels":{"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {manager Update v1 2022-11-09 01:04:55 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:cluster.x-k8s.io/cluster-name":{},"f:cluster.x-k8s.io/cluster-namespace":{},"f:cluster.x-k8s.io/machine":{},"f:cluster.x-k8s.io/owner-kind":{},"f:cluster.x-k8s.io/owner-name":{}}}} } {kube-controller-manager Update v1 2022-11-09 01:05:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:taints":{}}} } {Go-http-client Update v1 2022-11-09 01:05:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4VXLANTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-09 04:46:57 +0000 UTC FieldsV1 {"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:,DoNotUseExternalID:,ProviderID:azure:///subscriptions/===REDACTED===/resourceGroups/capz-conf-tvyqls/providers/Microsoft.Compute/virtualMachines/capz-conf-tvyqls-control-plane-7vbm8,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-azure-disk: {{4 0} {<nil>} 4 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{133003395072 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8344723456 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-azure-disk: {{4 0} {<nil>} 4 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{119703055367 0} {<nil>} 119703055367 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8239865856 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-09 01:05:27 +0000 UTC,LastTransitionTime:2022-11-09 01:05:27 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-09 04:46:57 +0000 UTC,LastTransitionTime:2022-11-09 01:04:17 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-09 04:46:57 +0000 UTC,LastTransitionTime:2022-11-09 01:04:17 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-09 04:46:57 +0000 UTC,LastTransitionTime:2022-11-09 01:04:17 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-09 04:46:57 +0000 UTC,LastTransitionTime:2022-11-09 01:05:22 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:Hostname,Address:capz-conf-tvyqls-control-plane-7vbm8,},NodeAddress{Type:InternalIP,Address:10.0.0.4,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:37a2e1ceaa0d455a8de0dda7f144eb7f,SystemUUID:50325a86-a564-f642-b7cf-2b4ce49e7328,BootID:49adcb05-103e-4672-b80c-d3eea5cf97ca,KernelVersion:5.4.0-1091-azure,OSImage:Ubuntu 18.04.6 LTS,ContainerRuntimeVersion:containerd://1.6.2,KubeletVersion:v1.26.0-alpha.3.425+2f2240400391ad,KubeProxyVersion:v1.26.0-alpha.3.425+2f2240400391ad,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[docker.io/calico/cni@sha256:914823d144204288f881e49b93b6852febfe669074cd4e2a782860981615f521 docker.io/calico/cni:v3.23.0],SizeBytes:110494683,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:b83c1d70989e1fe87583607bf5aee1ee34e52773d4755b95f5cf5a451962f3a4 registry.k8s.io/etcd:3.5.5-0],SizeBytes:102417044,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:102157811,},ContainerImage{Names:[docker.io/calico/node@sha256:4763820ecb4d8e82483a2ffabfec7fcded9603318692df210a778d223a4d7474 docker.io/calico/node:v3.23.0],SizeBytes:71573794,},ContainerImage{Names:[docker.io/calico/kube-controllers@sha256:78bc199299f966b0694dc4044501aee2d7ebd6862b2b0a00bca3ee8d3813c82f docker.io/calico/kube-controllers:v3.23.0],SizeBytes:56343954,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:51155161,},ContainerImage{Names:[gcr.io/k8s-staging-ci-images/kube-apiserver@sha256:f4a034110c7fecfbd53047bb4e7a568b1d123694426129781f718b2b83f650dd gcr.io/k8s-staging-ci-images/kube-apiserver:v1.26.0-alpha.3.425_2f2240400391ad],SizeBytes:34983033,},ContainerImage{Names:[registry.k8s.io/kube-apiserver@sha256:4188262a351f156e8027ff81693d771c35b34b668cbd61e59c4a4490dd5c08f3 registry.k8s.io/kube-apiserver:v1.25.3],SizeBytes:34238163,},ContainerImage{Names:[gcr.io/k8s-staging-ci-images/kube-controller-manager@sha256:4660397406f51cc97599ef65f46644d7cffad440bc3169fdfc0224626b3b2780 gcr.io/k8s-staging-ci-images/kube-controller-manager:v1.26.0-alpha.3.425_2f2240400391ad],SizeBytes:31999085,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager@sha256:d3a06262256f3e7578d5f77df137a8cdf58f9f498f35b5b56d116e8a7e31dc91 registry.k8s.io/kube-controller-manager:v1.25.3],SizeBytes:31261869,},ContainerImage{Names:[k8s.gcr.io/metrics-server/metrics-server@sha256:6385aec64bb97040a5e692947107b81e178555c7a5b71caa90d733e4130efc10 k8s.gcr.io/metrics-server/metrics-server:v0.5.2],SizeBytes:26023008,},ContainerImage{Names:[gcr.io/k8s-staging-ci-images/kube-proxy@sha256:8dcde74eedeada05ac0d0b3fec95ca4299544d64c362e7968f7f5ecb69186030 gcr.io/k8s-staging-ci-images/kube-proxy:v1.26.0-alpha.3.425_2f2240400391ad],SizeBytes:21362412,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:6bf25f038543e1f433cb7f2bdda445ed348c7b9279935ebc2ae4f432308ed82f registry.k8s.io/kube-proxy:v1.25.3],SizeBytes:20265805,},ContainerImage{Names:[gcr.io/k8s-staging-ci-images/kube-scheduler@sha256:fb955dc0dd068e8c525ac60042dac1c9cd349272855639394d761125851f8397 gcr.io/k8s-staging-ci-images/kube-scheduler:v1.26.0-alpha.3.425_2f2240400391ad],SizeBytes:17278982,},ContainerImage{Names:[registry.k8s.io/kube-scheduler@sha256:f478aa916568b00269068ff1e9ff742ecc16192eb6e371e30f69f75df904162e registry.k8s.io/kube-scheduler:v1.25.3],SizeBytes:15798744,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[docker.io/sigwindowstools/k8s-gmsa-webhook@sha256:56bba3b5a4788c05dc45f51444b924f6bed05a29789e04e43a6e2c3dbb07d9ce docker.io/sigwindowstools/k8s-gmsa-webhook:latest],SizeBytes:8874257,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Nov  9 04:50:23.602: INFO: 
Logging kubelet events for node capz-conf-tvyqls-control-plane-7vbm8
Nov  9 04:50:23.633: INFO: 
Logging pods the kubelet thinks is on node capz-conf-tvyqls-control-plane-7vbm8
Nov  9 04:50:23.689: INFO: kube-apiserver-capz-conf-tvyqls-control-plane-7vbm8 started at 2022-11-09 01:04:42 +0000 UTC (0+1 container statuses recorded)
Nov  9 04:50:23.689: INFO: 	Container kube-apiserver ready: true, restart count 0
Nov  9 04:50:23.689: INFO: kube-controller-manager-capz-conf-tvyqls-control-plane-7vbm8 started at 2022-11-09 01:04:42 +0000 UTC (0+1 container statuses recorded)
Nov  9 04:50:23.689: INFO: 	Container kube-controller-manager ready: true, restart count 0
Nov  9 04:50:23.689: INFO: kube-scheduler-capz-conf-tvyqls-control-plane-7vbm8 started at 2022-11-09 01:04:42 +0000 UTC (0+1 container statuses recorded)
Nov  9 04:50:23.689: INFO: 	Container kube-scheduler ready: true, restart count 0
Nov  9 04:50:23.689: INFO: kube-proxy-9ns2c started at 2022-11-09 01:04:42 +0000 UTC (0+1 container statuses recorded)
Nov  9 04:50:23.689: INFO: 	Container kube-proxy ready: true, restart count 0
Nov  9 04:50:23.689: INFO: calico-kube-controllers-56c5ff4bf8-gbkvc started at 2022-11-09 01:05:22 +0000 UTC (0+1 container statuses recorded)
Nov  9 04:50:23.689: INFO: 	Container calico-kube-controllers ready: true, restart count 0
Nov  9 04:50:23.689: INFO: metrics-server-954b56d74-7hmq4 started at 2022-11-09 01:05:30 +0000 UTC (0+1 container statuses recorded)
Nov  9 04:50:23.689: INFO: 	Container metrics-server ready: true, restart count 0
Nov  9 04:50:23.689: INFO: etcd-capz-conf-tvyqls-control-plane-7vbm8 started at 2022-11-09 01:04:42 +0000 UTC (0+1 container statuses recorded)
Nov  9 04:50:23.689: INFO: 	Container etcd ready: true, restart count 0
Nov  9 04:50:23.689: INFO: coredns-fdff55fb9-4wwnr started at 2022-11-09 01:10:12 +0000 UTC (0+1 container statuses recorded)
Nov  9 04:50:23.689: INFO: 	Container coredns ready: true, restart count 0
Nov  9 04:50:23.689: INFO: coredns-fdff55fb9-vl42q started at 2022-11-09 01:10:12 +0000 UTC (0+1 container statuses recorded)
Nov  9 04:50:23.689: INFO: 	Container coredns ready: true, restart count 0
Nov  9 04:50:23.689: INFO: calico-node-7bphx started at 2022-11-09 01:05:02 +0000 UTC (2+1 container statuses recorded)
Nov  9 04:50:23.689: INFO: 	Init container upgrade-ipam ready: true, restart count 0
Nov  9 04:50:23.689: INFO: 	Init container install-cni ready: true, restart count 0
Nov  9 04:50:23.689: INFO: 	Container calico-node ready: true, restart count 0
Nov  9 04:50:23.862: INFO: 
Latency metrics for node capz-conf-tvyqls-control-plane-7vbm8
[DeferCleanup (Each)] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: CPU)
  tear down framework | framework.go:193
STEP: Destroying namespace "horizontal-pod-autoscaling-2763" for this suite. 11/09/22 04:50:23.862

				Click to see stdout/stderrfrom junit_01.xml

Filter through log files | View test history on testgrid

Kubernetes e2e suite [It] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: CPU) [Serial] [Slow] ReplicaSet Should scale from 1 pod to 3 pods and then from 3 pods to 5 pods 15m40s

go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\s\[sig\-autoscaling\]\s\[Feature\:HPA\]\sHorizontal\spod\sautoscaling\s\(scale\sresource\:\sCPU\)\s\[Serial\]\s\[Slow\]\sReplicaSet\sShould\sscale\sfrom\s1\spod\sto\s3\spods\sand\sthen\sfrom\s3\spods\sto\s5\spods$'

test/e2e/autoscaling/horizontal_pod_autoscaling.go:209
k8s.io/kubernetes/test/e2e/autoscaling.(*HPAScaleTest).run(0xc001ed9e68, {0x74b75d8?, 0xc00275eba0?}, {{0x74b92e8, 0x4}, {0x74c2479, 0x7}, {0x74cb452, 0xa}}, 0xc00082cd20)
	test/e2e/autoscaling/horizontal_pod_autoscaling.go:209 +0x2d8
k8s.io/kubernetes/test/e2e/autoscaling.scaleUp({0x74b75d8?, 0x61dc905?}, {{0x74b92e8, 0x4}, {0x74c2479, 0x7}, {0x74cb452, 0xa}}, {0x74b8538, 0x3}, ...)
	test/e2e/autoscaling/horizontal_pod_autoscaling.go:249 +0x212
k8s.io/kubernetes/test/e2e/autoscaling.glob..func6.3.1()
	test/e2e/autoscaling/horizontal_pod_autoscaling.go:71 +0x88
				[BeforeEach] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: CPU)
  set up framework | framework.go:178
STEP: Creating a kubernetes client 11/09/22 02:27:15.867
Nov  9 02:27:15.867: INFO: >>> kubeConfig: /home/prow/go/src/k8s.io/windows-testing/capz-conf-tvyqls.kubeconfig
I1109 02:27:15.868111   14108 discovery.go:214] Invalidating discovery information
STEP: Building a namespace api object, basename horizontal-pod-autoscaling 11/09/22 02:27:15.868
I1109 02:27:15.905554   14108 reflector.go:221] Starting reflector *v1.ServiceAccount (0s) from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
I1109 02:27:15.905703   14108 reflector.go:257] Listing and watching *v1.ServiceAccount from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
I1109 02:27:15.968248   14108 reflector.go:227] Stopping reflector *v1.ServiceAccount (0s) from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
STEP: Waiting for a default service account to be provisioned in namespace 11/09/22 02:27:15.968
I1109 02:27:15.968456   14108 reflector.go:221] Starting reflector *v1.ServiceAccount (0s) from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
I1109 02:27:15.968470   14108 reflector.go:257] Listing and watching *v1.ServiceAccount from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
I1109 02:27:16.030248   14108 reflector.go:227] Stopping reflector *v1.ServiceAccount (0s) from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
STEP: Waiting for kube-root-ca.crt to be provisioned in namespace 11/09/22 02:27:16.03
I1109 02:27:16.030685   14108 reflector.go:221] Starting reflector *v1.ConfigMap (0s) from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
I1109 02:27:16.030706   14108 reflector.go:257] Listing and watching *v1.ConfigMap from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
I1109 02:27:16.092576   14108 reflector.go:227] Stopping reflector *v1.ConfigMap (0s) from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
[BeforeEach] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: CPU)
  test/e2e/framework/metrics/init/init.go:31
[It] Should scale from 1 pod to 3 pods and then from 3 pods to 5 pods
  test/e2e/autoscaling/horizontal_pod_autoscaling.go:70
Nov  9 02:27:16.092: INFO: >>> kubeConfig: /home/prow/go/src/k8s.io/windows-testing/capz-conf-tvyqls.kubeconfig
STEP: Running consuming RC rs via apps/v1beta2, Kind=ReplicaSet with 1 replicas 11/09/22 02:27:16.093
STEP: Creating replicaset rs in namespace horizontal-pod-autoscaling-1228 11/09/22 02:27:16.138
STEP: creating replicaset rs in namespace horizontal-pod-autoscaling-1228 11/09/22 02:27:16.138
I1109 02:27:16.175680   14108 runners.go:193] Created replica set with name: rs, namespace: horizontal-pod-autoscaling-1228, replica count: 1
I1109 02:27:16.175803   14108 reflector.go:221] Starting reflector *v1.Pod (0s) from test/utils/pod_store.go:57
I1109 02:27:16.175815   14108 reflector.go:257] Listing and watching *v1.Pod from test/utils/pod_store.go:57
I1109 02:27:26.227329   14108 runners.go:193] rs Pods: 1 out of 1 created, 1 running, 0 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
STEP: Running controller 11/09/22 02:27:26.227
I1109 02:27:26.227805   14108 reflector.go:227] Stopping reflector *v1.Pod (0s) from test/utils/pod_store.go:57
STEP: creating replication controller rs-ctrl in namespace horizontal-pod-autoscaling-1228 11/09/22 02:27:26.274
I1109 02:27:26.309948   14108 runners.go:193] Created replication controller with name: rs-ctrl, namespace: horizontal-pod-autoscaling-1228, replica count: 1
I1109 02:27:26.310082   14108 reflector.go:221] Starting reflector *v1.Pod (0s) from test/utils/pod_store.go:57
I1109 02:27:26.310104   14108 reflector.go:257] Listing and watching *v1.Pod from test/utils/pod_store.go:57
I1109 02:27:36.361131   14108 runners.go:193] rs-ctrl Pods: 1 out of 1 created, 1 running, 0 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1109 02:27:36.361383   14108 reflector.go:227] Stopping reflector *v1.Pod (0s) from test/utils/pod_store.go:57
Nov  9 02:27:41.362: INFO: Waiting for amount of service:rs-ctrl endpoints to be 1
Nov  9 02:27:41.396: INFO: RC rs: consume 250 millicores in total
Nov  9 02:27:41.397: INFO: RC rs: setting consumption to 250 millicores in total
Nov  9 02:27:41.397: INFO: RC rs: sending request to consume 250 millicores
Nov  9 02:27:41.397: INFO: RC rs: consume 0 MB in total
Nov  9 02:27:41.397: INFO: RC rs: disabling mem consumption
Nov  9 02:27:41.397: INFO: RC rs: consume custom metric 0 in total
Nov  9 02:27:41.397: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-1228/services/rs-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:27:41.397: INFO: RC rs: disabling consumption of custom metric QPS
Nov  9 02:27:41.467: INFO: waiting for 3 replicas (current: 1)
Nov  9 02:28:01.501: INFO: waiting for 3 replicas (current: 1)
Nov  9 02:28:11.468: INFO: RC rs: sending request to consume 250 millicores
Nov  9 02:28:11.468: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-1228/services/rs-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:28:21.500: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:28:41.501: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:28:41.520: INFO: RC rs: sending request to consume 250 millicores
Nov  9 02:28:41.520: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-1228/services/rs-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:29:01.501: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:29:11.572: INFO: RC rs: sending request to consume 250 millicores
Nov  9 02:29:11.572: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-1228/services/rs-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:29:21.501: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:29:41.501: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:29:41.639: INFO: RC rs: sending request to consume 250 millicores
Nov  9 02:29:41.639: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-1228/services/rs-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:30:01.500: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:30:11.690: INFO: RC rs: sending request to consume 250 millicores
Nov  9 02:30:11.690: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-1228/services/rs-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:30:21.501: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:30:41.501: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:30:41.747: INFO: RC rs: sending request to consume 250 millicores
Nov  9 02:30:41.747: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-1228/services/rs-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:31:01.501: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:31:11.796: INFO: RC rs: sending request to consume 250 millicores
Nov  9 02:31:11.796: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-1228/services/rs-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:31:21.501: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:31:41.503: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:31:41.851: INFO: RC rs: sending request to consume 250 millicores
Nov  9 02:31:41.851: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-1228/services/rs-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:32:01.502: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:32:11.900: INFO: RC rs: sending request to consume 250 millicores
Nov  9 02:32:11.900: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-1228/services/rs-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:32:21.501: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:32:41.501: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:32:41.954: INFO: RC rs: sending request to consume 250 millicores
Nov  9 02:32:41.954: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-1228/services/rs-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:33:01.501: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:33:12.001: INFO: RC rs: sending request to consume 250 millicores
Nov  9 02:33:12.002: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-1228/services/rs-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:33:21.501: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:33:41.502: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:33:42.053: INFO: RC rs: sending request to consume 250 millicores
Nov  9 02:33:42.053: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-1228/services/rs-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:34:01.502: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:34:12.101: INFO: RC rs: sending request to consume 250 millicores
Nov  9 02:34:12.101: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-1228/services/rs-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:34:21.500: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:34:41.504: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:34:42.150: INFO: RC rs: sending request to consume 250 millicores
Nov  9 02:34:42.150: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-1228/services/rs-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:35:01.501: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:35:12.202: INFO: RC rs: sending request to consume 250 millicores
Nov  9 02:35:12.202: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-1228/services/rs-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:35:21.500: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:35:41.501: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:35:42.255: INFO: RC rs: sending request to consume 250 millicores
Nov  9 02:35:42.255: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-1228/services/rs-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:36:01.501: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:36:12.303: INFO: RC rs: sending request to consume 250 millicores
Nov  9 02:36:12.303: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-1228/services/rs-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:36:21.501: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:36:41.501: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:36:42.352: INFO: RC rs: sending request to consume 250 millicores
Nov  9 02:36:42.352: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-1228/services/rs-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:37:01.501: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:37:12.408: INFO: RC rs: sending request to consume 250 millicores
Nov  9 02:37:12.408: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-1228/services/rs-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:37:21.501: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:37:41.500: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:37:42.457: INFO: RC rs: sending request to consume 250 millicores
Nov  9 02:37:42.458: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-1228/services/rs-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:38:01.501: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:38:12.512: INFO: RC rs: sending request to consume 250 millicores
Nov  9 02:38:12.512: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-1228/services/rs-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:38:21.500: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:38:41.501: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:38:42.566: INFO: RC rs: sending request to consume 250 millicores
Nov  9 02:38:42.566: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-1228/services/rs-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:39:01.502: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:39:12.614: INFO: RC rs: sending request to consume 250 millicores
Nov  9 02:39:12.614: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-1228/services/rs-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:39:21.503: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:39:41.501: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:39:42.658: INFO: RC rs: sending request to consume 250 millicores
Nov  9 02:39:42.659: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-1228/services/rs-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:40:01.501: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:40:12.711: INFO: RC rs: sending request to consume 250 millicores
Nov  9 02:40:12.711: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-1228/services/rs-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:40:21.500: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:40:41.501: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:40:42.758: INFO: RC rs: sending request to consume 250 millicores
Nov  9 02:40:42.758: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-1228/services/rs-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:41:01.500: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:41:12.805: INFO: RC rs: sending request to consume 250 millicores
Nov  9 02:41:12.805: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-1228/services/rs-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:41:21.501: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:41:41.503: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:41:42.860: INFO: RC rs: sending request to consume 250 millicores
Nov  9 02:41:42.861: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-1228/services/rs-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:42:01.500: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:42:12.906: INFO: RC rs: sending request to consume 250 millicores
Nov  9 02:42:12.906: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-1228/services/rs-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:42:21.501: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:42:41.501: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:42:41.534: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:42:41.534: INFO: Unexpected error: timeout waiting 15m0s for 3 replicas: 
    <*errors.errorString | 0xc000287c80>: {
        s: "timed out waiting for the condition",
    }
Nov  9 02:42:41.534: FAIL: timeout waiting 15m0s for 3 replicas: timed out waiting for the condition

Full Stack Trace
k8s.io/kubernetes/test/e2e/autoscaling.(*HPAScaleTest).run(0xc001ed9e68, {0x74b75d8?, 0xc00275eba0?}, {{0x74b92e8, 0x4}, {0x74c2479, 0x7}, {0x74cb452, 0xa}}, 0xc00082cd20)
	test/e2e/autoscaling/horizontal_pod_autoscaling.go:209 +0x2d8
k8s.io/kubernetes/test/e2e/autoscaling.scaleUp({0x74b75d8?, 0x61dc905?}, {{0x74b92e8, 0x4}, {0x74c2479, 0x7}, {0x74cb452, 0xa}}, {0x74b8538, 0x3}, ...)
	test/e2e/autoscaling/horizontal_pod_autoscaling.go:249 +0x212
k8s.io/kubernetes/test/e2e/autoscaling.glob..func6.3.1()
	test/e2e/autoscaling/horizontal_pod_autoscaling.go:71 +0x88
STEP: Removing consuming RC rs 11/09/22 02:42:41.57
Nov  9 02:42:41.571: INFO: RC rs: stopping metric consumer
Nov  9 02:42:41.571: INFO: RC rs: stopping CPU consumer
Nov  9 02:42:41.571: INFO: RC rs: stopping mem consumer
STEP: deleting ReplicaSet.apps rs in namespace horizontal-pod-autoscaling-1228, will wait for the garbage collector to delete the pods 11/09/22 02:42:51.571
I1109 02:42:51.604884   14108 reflector.go:221] Starting reflector *v1.Pod (0s) from test/utils/pod_store.go:57
I1109 02:42:51.604915   14108 reflector.go:257] Listing and watching *v1.Pod from test/utils/pod_store.go:57
Nov  9 02:42:51.690: INFO: Deleting ReplicaSet.apps rs took: 34.805448ms
I1109 02:42:51.790804   14108 resource.go:655] Ignoring inactive pod horizontal-pod-autoscaling-1228/rs-wcdzt in state Running, deletion time 2022-11-09 02:43:21 +0000 UTC
I1109 02:42:51.790857   14108 resource.go:655] Ignoring inactive pod horizontal-pod-autoscaling-1228/rs-gj9zf in state Running, deletion time 2022-11-09 02:43:21 +0000 UTC
Nov  9 02:42:51.790: INFO: Terminating ReplicaSet.apps rs pods took: 100.262129ms
I1109 02:42:53.692002   14108 reflector.go:227] Stopping reflector *v1.Pod (0s) from test/utils/pod_store.go:57
STEP: deleting ReplicationController rs-ctrl in namespace horizontal-pod-autoscaling-1228, will wait for the garbage collector to delete the pods 11/09/22 02:42:53.752
I1109 02:42:53.785430   14108 reflector.go:221] Starting reflector *v1.Pod (0s) from test/utils/pod_store.go:57
I1109 02:42:53.785483   14108 reflector.go:257] Listing and watching *v1.Pod from test/utils/pod_store.go:57
Nov  9 02:42:53.872: INFO: Deleting ReplicationController rs-ctrl took: 36.719692ms
I1109 02:42:53.974191   14108 resource.go:655] Ignoring inactive pod horizontal-pod-autoscaling-1228/rs-ctrl-ms49z in state Running, deletion time 2022-11-09 02:42:54 +0000 UTC
Nov  9 02:42:53.974: INFO: Terminating ReplicationController rs-ctrl pods took: 101.337957ms
I1109 02:42:55.675621   14108 reflector.go:227] Stopping reflector *v1.Pod (0s) from test/utils/pod_store.go:57
[AfterEach] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: CPU)
  test/e2e/framework/node/init/init.go:32
Nov  9 02:42:55.735: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
[DeferCleanup (Each)] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: CPU)
  test/e2e/framework/metrics/init/init.go:33
[DeferCleanup (Each)] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: CPU)
  dump namespaces | framework.go:196
STEP: dump namespace information after failure 11/09/22 02:42:55.77
STEP: Collecting events from namespace "horizontal-pod-autoscaling-1228". 11/09/22 02:42:55.77
STEP: Found 19 events. 11/09/22 02:42:55.804
Nov  9 02:42:55.804: INFO: At 2022-11-09 02:27:16 +0000 UTC - event for rs: {replicaset-controller } SuccessfulCreate: Created pod: rs-wcdzt
Nov  9 02:42:55.804: INFO: At 2022-11-09 02:27:16 +0000 UTC - event for rs-wcdzt: {default-scheduler } Scheduled: Successfully assigned horizontal-pod-autoscaling-1228/rs-wcdzt to capz-conf-hhzhh
Nov  9 02:42:55.804: INFO: At 2022-11-09 02:27:20 +0000 UTC - event for rs-wcdzt: {kubelet capz-conf-hhzhh} Created: Created container rs
Nov  9 02:42:55.804: INFO: At 2022-11-09 02:27:20 +0000 UTC - event for rs-wcdzt: {kubelet capz-conf-hhzhh} Pulled: Container image "registry.k8s.io/e2e-test-images/resource-consumer:1.13" already present on machine
Nov  9 02:42:55.804: INFO: At 2022-11-09 02:27:23 +0000 UTC - event for rs-wcdzt: {kubelet capz-conf-hhzhh} Started: Started container rs
Nov  9 02:42:55.804: INFO: At 2022-11-09 02:27:26 +0000 UTC - event for rs-ctrl: {replication-controller } SuccessfulCreate: Created pod: rs-ctrl-ms49z
Nov  9 02:42:55.804: INFO: At 2022-11-09 02:27:26 +0000 UTC - event for rs-ctrl-ms49z: {default-scheduler } Scheduled: Successfully assigned horizontal-pod-autoscaling-1228/rs-ctrl-ms49z to capz-conf-m97cx
Nov  9 02:42:55.804: INFO: At 2022-11-09 02:27:30 +0000 UTC - event for rs-ctrl-ms49z: {kubelet capz-conf-m97cx} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.40" already present on machine
Nov  9 02:42:55.804: INFO: At 2022-11-09 02:27:30 +0000 UTC - event for rs-ctrl-ms49z: {kubelet capz-conf-m97cx} Created: Created container rs-ctrl
Nov  9 02:42:55.804: INFO: At 2022-11-09 02:27:32 +0000 UTC - event for rs-ctrl-ms49z: {kubelet capz-conf-m97cx} Started: Started container rs-ctrl
Nov  9 02:42:55.804: INFO: At 2022-11-09 02:27:56 +0000 UTC - event for rs: {horizontal-pod-autoscaler } SuccessfulRescale: New size: 2; reason: cpu resource utilization (percentage of request) above target
Nov  9 02:42:55.804: INFO: At 2022-11-09 02:27:56 +0000 UTC - event for rs: {replicaset-controller } SuccessfulCreate: Created pod: rs-gj9zf
Nov  9 02:42:55.804: INFO: At 2022-11-09 02:27:56 +0000 UTC - event for rs-gj9zf: {default-scheduler } Scheduled: Successfully assigned horizontal-pod-autoscaling-1228/rs-gj9zf to capz-conf-m97cx
Nov  9 02:42:55.804: INFO: At 2022-11-09 02:28:00 +0000 UTC - event for rs-gj9zf: {kubelet capz-conf-m97cx} Created: Created container rs
Nov  9 02:42:55.804: INFO: At 2022-11-09 02:28:00 +0000 UTC - event for rs-gj9zf: {kubelet capz-conf-m97cx} Pulled: Container image "registry.k8s.io/e2e-test-images/resource-consumer:1.13" already present on machine
Nov  9 02:42:55.804: INFO: At 2022-11-09 02:28:03 +0000 UTC - event for rs-gj9zf: {kubelet capz-conf-m97cx} Started: Started container rs
Nov  9 02:42:55.804: INFO: At 2022-11-09 02:42:51 +0000 UTC - event for rs-gj9zf: {kubelet capz-conf-m97cx} Killing: Stopping container rs
Nov  9 02:42:55.804: INFO: At 2022-11-09 02:42:51 +0000 UTC - event for rs-wcdzt: {kubelet capz-conf-hhzhh} Killing: Stopping container rs
Nov  9 02:42:55.804: INFO: At 2022-11-09 02:42:53 +0000 UTC - event for rs-ctrl-ms49z: {kubelet capz-conf-m97cx} Killing: Stopping container rs-ctrl
Nov  9 02:42:55.836: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
Nov  9 02:42:55.836: INFO: 
Nov  9 02:42:55.874: INFO: 
Logging node info for node capz-conf-hhzhh
Nov  9 02:42:55.909: INFO: Node Info: &Node{ObjectMeta:{capz-conf-hhzhh    641b6838-eeea-48b0-9dc0-0ff9ab096f10 12589 0 2022-11-09 01:07:16 +0000 UTC <nil> <nil> map[agentpool:windowsgmsa beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:windows failure-domain.beta.kubernetes.io/region:eastus failure-domain.beta.kubernetes.io/zone:0 kubernetes.io/arch:amd64 kubernetes.io/hostname:capz-conf-hhzhh kubernetes.io/os:windows node.kubernetes.io/instance-type:Standard_D4s_v3 node.kubernetes.io/windows-build:10.0.20348 topology.kubernetes.io/region:eastus topology.kubernetes.io/zone:0] map[cluster.x-k8s.io/cluster-name:capz-conf-tvyqls cluster.x-k8s.io/cluster-namespace:default cluster.x-k8s.io/machine:capz-conf-tvyqls-md-win-7848cfdb84-9mln2 cluster.x-k8s.io/owner-kind:MachineSet cluster.x-k8s.io/owner-name:capz-conf-tvyqls-md-win-7848cfdb84 kubeadm.alpha.kubernetes.io/cri-socket:npipe:////./pipe/containerd-containerd node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:10.1.0.4/16 projectcalico.org/IPv4VXLANTunnelAddr:192.168.84.65 projectcalico.org/VXLANTunnelMACAddr:00:15:5d:5b:b6:ce volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubeadm Update v1 2022-11-09 01:07:16 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:kubeadm.alpha.kubernetes.io/cri-socket":{}}}} } {kubelet.exe Update v1 2022-11-09 01:07:16 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:node.kubernetes.io/windows-build":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {manager Update v1 2022-11-09 01:08:02 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:cluster.x-k8s.io/cluster-name":{},"f:cluster.x-k8s.io/cluster-namespace":{},"f:cluster.x-k8s.io/machine":{},"f:cluster.x-k8s.io/owner-kind":{},"f:cluster.x-k8s.io/owner-name":{}}}} } {kube-controller-manager Update v1 2022-11-09 01:08:06 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}}} } {Go-http-client Update v1 2022-11-09 01:08:32 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4VXLANTunnelAddr":{},"f:projectcalico.org/VXLANTunnelMACAddr":{}}}} status} {cluster-api-e2e Update v1 2022-11-09 01:10:08 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:agentpool":{}}}} } {e2e.test Update v1 2022-11-09 01:22:08 +0000 UTC FieldsV1 {"f:status":{"f:capacity":{"f:example.com/fakePTSRes":{}}}} status} {kubelet.exe Update v1 2022-11-09 02:38:49 +0000 UTC FieldsV1 {"f:status":{"f:allocatable":{"f:example.com/fakePTSRes":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:,DoNotUseExternalID:,ProviderID:azure:///subscriptions/===REDACTED===/resourceGroups/capz-conf-tvyqls/providers/Microsoft.Compute/virtualMachines/capz-conf-hhzhh,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-azure-disk: {{8 0} {<nil>} 8 DecimalSI},cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{136912564224 0} {<nil>} 133703676Ki BinarySI},example.com/fakePTSRes: {{10 0} {<nil>} 10 DecimalSI},memory: {{17179398144 0} {<nil>} 16776756Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-azure-disk: {{8 0} {<nil>} 8 DecimalSI},cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{123221307598 0} {<nil>} 123221307598 DecimalSI},example.com/fakePTSRes: {{10 0} {<nil>} 10 DecimalSI},memory: {{17074540544 0} {<nil>} 16674356Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-09 02:38:49 +0000 UTC,LastTransitionTime:2022-11-09 01:07:16 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-09 02:38:49 +0000 UTC,LastTransitionTime:2022-11-09 01:07:16 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-09 02:38:49 +0000 UTC,LastTransitionTime:2022-11-09 01:07:16 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-09 02:38:49 +0000 UTC,LastTransitionTime:2022-11-09 01:08:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status,},},Addresses:[]NodeAddress{NodeAddress{Type:Hostname,Address:capz-conf-hhzhh,},NodeAddress{Type:InternalIP,Address:10.1.0.4,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:capz-conf-hhzhh,SystemUUID:5F9EA3E7-22F1-406E-9AAC-69BA5DD83BF4,BootID:8,KernelVersion:10.0.20348.1129,OSImage:Windows Server 2022 Datacenter,ContainerRuntimeVersion:containerd://1.6.8,KubeletVersion:v1.26.0-alpha.3.425+2f2240400391ad,KubeProxyVersion:v1.26.0-alpha.3.425+2f2240400391ad,OperatingSystem:windows,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:307641217,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:221349084,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:218942479,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:183620591,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:181784786,},ContainerImage{Names:[ghcr.io/kubernetes-sigs/sig-windows/eventflow-logger@sha256:63bf2aa9db909d0d90fb5205abf7fb2a6d9a494b89cbd2508a42457dfc875505 ghcr.io/kubernetes-sigs/sig-windows/eventflow-logger:v0.1.0],SizeBytes:148054279,},ContainerImage{Names:[ghcr.io/kubernetes-sigs/sig-windows/csi-proxy@sha256:96b4144986319a747ba599892454be2737aae6005d96b8e13ed481321ac3afba ghcr.io/kubernetes-sigs/sig-windows/csi-proxy:v1.0.2],SizeBytes:123925937,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/resource-consumer@sha256:ba5e047a337e5d0709bc57df45b95b2c7f6f2794b290e4e24f7fc8980d60b25a registry.k8s.io/e2e-test-images/resource-consumer:1.13],SizeBytes:121182992,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6],SizeBytes:119397646,},ContainerImage{Names:[docker.io/sigwindowstools/kube-proxy:v1.23.1-calico-hostprocess docker.io/sigwindowstools/kube-proxy:v1.26.0-alpha.3.425_2f2240400391ad-calico-hostprocess],SizeBytes:116182072,},ContainerImage{Names:[docker.io/sigwindowstools/calico-install@sha256:97bc10aa5000a0ee1c842ac32771fe7a45a3a5ca507711bdf57ae2eb5f293e2b docker.io/sigwindowstools/calico-install:v3.23.0-hostprocess],SizeBytes:47258343,},ContainerImage{Names:[docker.io/sigwindowstools/calico-node@sha256:ea8b55bde9aed6a649582a6e21029577430661c743d94b3a5e93d57e648874a2 docker.io/sigwindowstools/calico-node:v3.23.0-hostprocess],SizeBytes:27005624,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Nov  9 02:42:55.909: INFO: 
Logging kubelet events for node capz-conf-hhzhh
Nov  9 02:42:55.941: INFO: 
Logging pods the kubelet thinks is on node capz-conf-hhzhh
Nov  9 02:42:55.993: INFO: containerd-logger-z56l7 started at 2022-11-09 01:07:16 +0000 UTC (0+1 container statuses recorded)
Nov  9 02:42:55.993: INFO: 	Container containerd-logger ready: true, restart count 0
Nov  9 02:42:55.993: INFO: csi-proxy-dkn8s started at 2022-11-09 01:08:06 +0000 UTC (0+1 container statuses recorded)
Nov  9 02:42:55.993: INFO: 	Container csi-proxy ready: true, restart count 0
Nov  9 02:42:55.993: INFO: kube-proxy-windows-ldnh5 started at 2022-11-09 01:07:16 +0000 UTC (0+1 container statuses recorded)
Nov  9 02:42:55.993: INFO: 	Container kube-proxy ready: true, restart count 0
Nov  9 02:42:55.993: INFO: calico-node-windows-jpm48 started at 2022-11-09 01:07:16 +0000 UTC (1+2 container statuses recorded)
Nov  9 02:42:55.993: INFO: 	Init container install-cni ready: true, restart count 0
Nov  9 02:42:55.993: INFO: 	Container calico-node-felix ready: true, restart count 1
Nov  9 02:42:55.993: INFO: 	Container calico-node-startup ready: true, restart count 0
Nov  9 02:42:56.191: INFO: 
Latency metrics for node capz-conf-hhzhh
Nov  9 02:42:56.191: INFO: 
Logging node info for node capz-conf-m97cx
Nov  9 02:42:56.225: INFO: Node Info: &Node{ObjectMeta:{capz-conf-m97cx    36965107-96dc-426b-924d-c291c94a2d04 12575 0 2022-11-09 01:07:00 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:windows failure-domain.beta.kubernetes.io/region:eastus failure-domain.beta.kubernetes.io/zone:0 kubernetes.io/arch:amd64 kubernetes.io/hostname:capz-conf-m97cx kubernetes.io/os:windows node.kubernetes.io/instance-type:Standard_D4s_v3 node.kubernetes.io/windows-build:10.0.20348 topology.kubernetes.io/region:eastus topology.kubernetes.io/zone:0] map[cluster.x-k8s.io/cluster-name:capz-conf-tvyqls cluster.x-k8s.io/cluster-namespace:default cluster.x-k8s.io/machine:capz-conf-tvyqls-md-win-7848cfdb84-cw8bk cluster.x-k8s.io/owner-kind:MachineSet cluster.x-k8s.io/owner-name:capz-conf-tvyqls-md-win-7848cfdb84 kubeadm.alpha.kubernetes.io/cri-socket:npipe:////./pipe/containerd-containerd node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:10.1.0.5/16 projectcalico.org/IPv4VXLANTunnelAddr:192.168.100.1 projectcalico.org/VXLANTunnelMACAddr:00:15:5d:ac:92:68 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubeadm Update v1 2022-11-09 01:07:00 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:kubeadm.alpha.kubernetes.io/cri-socket":{}}}} } {kubelet.exe Update v1 2022-11-09 01:07:00 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:node.kubernetes.io/windows-build":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-11-09 01:07:31 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}}} } {manager Update v1 2022-11-09 01:08:02 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:cluster.x-k8s.io/cluster-name":{},"f:cluster.x-k8s.io/cluster-namespace":{},"f:cluster.x-k8s.io/machine":{},"f:cluster.x-k8s.io/owner-kind":{},"f:cluster.x-k8s.io/owner-name":{}}}} } {Go-http-client Update v1 2022-11-09 01:08:08 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4VXLANTunnelAddr":{},"f:projectcalico.org/VXLANTunnelMACAddr":{}}}} status} {e2e.test Update v1 2022-11-09 01:22:07 +0000 UTC FieldsV1 {"f:status":{"f:capacity":{"f:example.com/fakePTSRes":{}}}} status} {kubelet.exe Update v1 2022-11-09 02:38:41 +0000 UTC FieldsV1 {"f:status":{"f:allocatable":{"f:example.com/fakePTSRes":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:,DoNotUseExternalID:,ProviderID:azure:///subscriptions/===REDACTED===/resourceGroups/capz-conf-tvyqls/providers/Microsoft.Compute/virtualMachines/capz-conf-m97cx,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-azure-disk: {{8 0} {<nil>} 8 DecimalSI},cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{136912564224 0} {<nil>} 133703676Ki BinarySI},example.com/fakePTSRes: {{10 0} {<nil>} 10 DecimalSI},memory: {{17179398144 0} {<nil>} 16776756Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-azure-disk: {{8 0} {<nil>} 8 DecimalSI},cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{123221307598 0} {<nil>} 123221307598 DecimalSI},example.com/fakePTSRes: {{10 0} {<nil>} 10 DecimalSI},memory: {{17074540544 0} {<nil>} 16674356Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-09 02:38:41 +0000 UTC,LastTransitionTime:2022-11-09 01:07:00 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-09 02:38:41 +0000 UTC,LastTransitionTime:2022-11-09 01:07:00 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-09 02:38:41 +0000 UTC,LastTransitionTime:2022-11-09 01:07:00 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-09 02:38:41 +0000 UTC,LastTransitionTime:2022-11-09 01:07:31 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status,},},Addresses:[]NodeAddress{NodeAddress{Type:Hostname,Address:capz-conf-m97cx,},NodeAddress{Type:InternalIP,Address:10.1.0.5,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:capz-conf-m97cx,SystemUUID:FC504DDA-1329-4BBA-8C1E-77FAD783BCA1,BootID:8,KernelVersion:10.0.20348.1129,OSImage:Windows Server 2022 Datacenter,ContainerRuntimeVersion:containerd://1.6.8,KubeletVersion:v1.26.0-alpha.3.425+2f2240400391ad,KubeProxyVersion:v1.26.0-alpha.3.425+2f2240400391ad,OperatingSystem:windows,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:307641217,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:221349084,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:218942479,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:183620591,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:181784786,},ContainerImage{Names:[ghcr.io/kubernetes-sigs/sig-windows/eventflow-logger@sha256:63bf2aa9db909d0d90fb5205abf7fb2a6d9a494b89cbd2508a42457dfc875505 ghcr.io/kubernetes-sigs/sig-windows/eventflow-logger:v0.1.0],SizeBytes:148054279,},ContainerImage{Names:[ghcr.io/kubernetes-sigs/sig-windows/csi-proxy@sha256:96b4144986319a747ba599892454be2737aae6005d96b8e13ed481321ac3afba ghcr.io/kubernetes-sigs/sig-windows/csi-proxy:v1.0.2],SizeBytes:123925937,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/resource-consumer@sha256:ba5e047a337e5d0709bc57df45b95b2c7f6f2794b290e4e24f7fc8980d60b25a registry.k8s.io/e2e-test-images/resource-consumer:1.13],SizeBytes:121182992,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6],SizeBytes:119397646,},ContainerImage{Names:[docker.io/sigwindowstools/kube-proxy:v1.23.1-calico-hostprocess docker.io/sigwindowstools/kube-proxy:v1.26.0-alpha.3.425_2f2240400391ad-calico-hostprocess],SizeBytes:116182072,},ContainerImage{Names:[docker.io/sigwindowstools/calico-install@sha256:97bc10aa5000a0ee1c842ac32771fe7a45a3a5ca507711bdf57ae2eb5f293e2b docker.io/sigwindowstools/calico-install:v3.23.0-hostprocess],SizeBytes:47258343,},ContainerImage{Names:[docker.io/sigwindowstools/calico-node@sha256:ea8b55bde9aed6a649582a6e21029577430661c743d94b3a5e93d57e648874a2 docker.io/sigwindowstools/calico-node:v3.23.0-hostprocess],SizeBytes:27005624,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Nov  9 02:42:56.226: INFO: 
Logging kubelet events for node capz-conf-m97cx
Nov  9 02:42:56.258: INFO: 
Logging pods the kubelet thinks is on node capz-conf-m97cx
Nov  9 02:42:56.311: INFO: kube-proxy-windows-55jdx started at 2022-11-09 01:07:01 +0000 UTC (0+1 container statuses recorded)
Nov  9 02:42:56.311: INFO: 	Container kube-proxy ready: true, restart count 0
Nov  9 02:42:56.311: INFO: csi-proxy-tn9x7 started at 2022-11-09 01:07:31 +0000 UTC (0+1 container statuses recorded)
Nov  9 02:42:56.311: INFO: 	Container csi-proxy ready: true, restart count 0
Nov  9 02:42:56.311: INFO: calico-node-windows-w5q9w started at 2022-11-09 01:07:01 +0000 UTC (1+2 container statuses recorded)
Nov  9 02:42:56.311: INFO: 	Init container install-cni ready: true, restart count 0
Nov  9 02:42:56.311: INFO: 	Container calico-node-felix ready: true, restart count 1
Nov  9 02:42:56.311: INFO: 	Container calico-node-startup ready: true, restart count 0
Nov  9 02:42:56.311: INFO: containerd-logger-v9qh5 started at 2022-11-09 01:07:01 +0000 UTC (0+1 container statuses recorded)
Nov  9 02:42:56.311: INFO: 	Container containerd-logger ready: true, restart count 0
Nov  9 02:42:56.487: INFO: 
Latency metrics for node capz-conf-m97cx
Nov  9 02:42:56.487: INFO: 
Logging node info for node capz-conf-tvyqls-control-plane-7vbm8
Nov  9 02:42:56.522: INFO: Node Info: &Node{ObjectMeta:{capz-conf-tvyqls-control-plane-7vbm8    59067037-dec8-4e79-8a8a-d150ea7ce6ba 12649 0 2022-11-09 01:04:38 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D2s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eastus failure-domain.beta.kubernetes.io/zone:eastus-3 kubernetes.io/arch:amd64 kubernetes.io/hostname:capz-conf-tvyqls-control-plane-7vbm8 kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:Standard_D2s_v3 topology.kubernetes.io/region:eastus topology.kubernetes.io/zone:eastus-3] map[cluster.x-k8s.io/cluster-name:capz-conf-tvyqls cluster.x-k8s.io/cluster-namespace:default cluster.x-k8s.io/machine:capz-conf-tvyqls-control-plane-ljqrh cluster.x-k8s.io/owner-kind:KubeadmControlPlane cluster.x-k8s.io/owner-name:capz-conf-tvyqls-control-plane kubeadm.alpha.kubernetes.io/cri-socket:unix:///var/run/containerd/containerd.sock node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:10.0.0.4/16 projectcalico.org/IPv4VXLANTunnelAddr:192.168.2.192 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-11-09 01:04:38 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kubeadm Update v1 2022-11-09 01:04:40 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:kubeadm.alpha.kubernetes.io/cri-socket":{}},"f:labels":{"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {manager Update v1 2022-11-09 01:04:55 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:cluster.x-k8s.io/cluster-name":{},"f:cluster.x-k8s.io/cluster-namespace":{},"f:cluster.x-k8s.io/machine":{},"f:cluster.x-k8s.io/owner-kind":{},"f:cluster.x-k8s.io/owner-name":{}}}} } {kube-controller-manager Update v1 2022-11-09 01:05:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:taints":{}}} } {Go-http-client Update v1 2022-11-09 01:05:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4VXLANTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-09 02:39:26 +0000 UTC FieldsV1 {"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:,DoNotUseExternalID:,ProviderID:azure:///subscriptions/===REDACTED===/resourceGroups/capz-conf-tvyqls/providers/Microsoft.Compute/virtualMachines/capz-conf-tvyqls-control-plane-7vbm8,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-azure-disk: {{4 0} {<nil>} 4 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{133003395072 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8344723456 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-azure-disk: {{4 0} {<nil>} 4 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{119703055367 0} {<nil>} 119703055367 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8239865856 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-09 01:05:27 +0000 UTC,LastTransitionTime:2022-11-09 01:05:27 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-09 02:39:26 +0000 UTC,LastTransitionTime:2022-11-09 01:04:17 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-09 02:39:26 +0000 UTC,LastTransitionTime:2022-11-09 01:04:17 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-09 02:39:26 +0000 UTC,LastTransitionTime:2022-11-09 01:04:17 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-09 02:39:26 +0000 UTC,LastTransitionTime:2022-11-09 01:05:22 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:Hostname,Address:capz-conf-tvyqls-control-plane-7vbm8,},NodeAddress{Type:InternalIP,Address:10.0.0.4,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:37a2e1ceaa0d455a8de0dda7f144eb7f,SystemUUID:50325a86-a564-f642-b7cf-2b4ce49e7328,BootID:49adcb05-103e-4672-b80c-d3eea5cf97ca,KernelVersion:5.4.0-1091-azure,OSImage:Ubuntu 18.04.6 LTS,ContainerRuntimeVersion:containerd://1.6.2,KubeletVersion:v1.26.0-alpha.3.425+2f2240400391ad,KubeProxyVersion:v1.26.0-alpha.3.425+2f2240400391ad,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[docker.io/calico/cni@sha256:914823d144204288f881e49b93b6852febfe669074cd4e2a782860981615f521 docker.io/calico/cni:v3.23.0],SizeBytes:110494683,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:b83c1d70989e1fe87583607bf5aee1ee34e52773d4755b95f5cf5a451962f3a4 registry.k8s.io/etcd:3.5.5-0],SizeBytes:102417044,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:102157811,},ContainerImage{Names:[docker.io/calico/node@sha256:4763820ecb4d8e82483a2ffabfec7fcded9603318692df210a778d223a4d7474 docker.io/calico/node:v3.23.0],SizeBytes:71573794,},ContainerImage{Names:[docker.io/calico/kube-controllers@sha256:78bc199299f966b0694dc4044501aee2d7ebd6862b2b0a00bca3ee8d3813c82f docker.io/calico/kube-controllers:v3.23.0],SizeBytes:56343954,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:51155161,},ContainerImage{Names:[gcr.io/k8s-staging-ci-images/kube-apiserver@sha256:f4a034110c7fecfbd53047bb4e7a568b1d123694426129781f718b2b83f650dd gcr.io/k8s-staging-ci-images/kube-apiserver:v1.26.0-alpha.3.425_2f2240400391ad],SizeBytes:34983033,},ContainerImage{Names:[registry.k8s.io/kube-apiserver@sha256:4188262a351f156e8027ff81693d771c35b34b668cbd61e59c4a4490dd5c08f3 registry.k8s.io/kube-apiserver:v1.25.3],SizeBytes:34238163,},ContainerImage{Names:[gcr.io/k8s-staging-ci-images/kube-controller-manager@sha256:4660397406f51cc97599ef65f46644d7cffad440bc3169fdfc0224626b3b2780 gcr.io/k8s-staging-ci-images/kube-controller-manager:v1.26.0-alpha.3.425_2f2240400391ad],SizeBytes:31999085,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager@sha256:d3a06262256f3e7578d5f77df137a8cdf58f9f498f35b5b56d116e8a7e31dc91 registry.k8s.io/kube-controller-manager:v1.25.3],SizeBytes:31261869,},ContainerImage{Names:[k8s.gcr.io/metrics-server/metrics-server@sha256:6385aec64bb97040a5e692947107b81e178555c7a5b71caa90d733e4130efc10 k8s.gcr.io/metrics-server/metrics-server:v0.5.2],SizeBytes:26023008,},ContainerImage{Names:[gcr.io/k8s-staging-ci-images/kube-proxy@sha256:8dcde74eedeada05ac0d0b3fec95ca4299544d64c362e7968f7f5ecb69186030 gcr.io/k8s-staging-ci-images/kube-proxy:v1.26.0-alpha.3.425_2f2240400391ad],SizeBytes:21362412,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:6bf25f038543e1f433cb7f2bdda445ed348c7b9279935ebc2ae4f432308ed82f registry.k8s.io/kube-proxy:v1.25.3],SizeBytes:20265805,},ContainerImage{Names:[gcr.io/k8s-staging-ci-images/kube-scheduler@sha256:fb955dc0dd068e8c525ac60042dac1c9cd349272855639394d761125851f8397 gcr.io/k8s-staging-ci-images/kube-scheduler:v1.26.0-alpha.3.425_2f2240400391ad],SizeBytes:17278982,},ContainerImage{Names:[registry.k8s.io/kube-scheduler@sha256:f478aa916568b00269068ff1e9ff742ecc16192eb6e371e30f69f75df904162e registry.k8s.io/kube-scheduler:v1.25.3],SizeBytes:15798744,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[docker.io/sigwindowstools/k8s-gmsa-webhook@sha256:56bba3b5a4788c05dc45f51444b924f6bed05a29789e04e43a6e2c3dbb07d9ce docker.io/sigwindowstools/k8s-gmsa-webhook:latest],SizeBytes:8874257,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Nov  9 02:42:56.523: INFO: 
Logging kubelet events for node capz-conf-tvyqls-control-plane-7vbm8
Nov  9 02:42:56.555: INFO: 
Logging pods the kubelet thinks is on node capz-conf-tvyqls-control-plane-7vbm8
Nov  9 02:42:56.609: INFO: calico-node-7bphx started at 2022-11-09 01:05:02 +0000 UTC (2+1 container statuses recorded)
Nov  9 02:42:56.609: INFO: 	Init container upgrade-ipam ready: true, restart count 0
Nov  9 02:42:56.609: INFO: 	Init container install-cni ready: true, restart count 0
Nov  9 02:42:56.609: INFO: 	Container calico-node ready: true, restart count 0
Nov  9 02:42:56.609: INFO: coredns-fdff55fb9-vl42q started at 2022-11-09 01:10:12 +0000 UTC (0+1 container statuses recorded)
Nov  9 02:42:56.609: INFO: 	Container coredns ready: true, restart count 0
Nov  9 02:42:56.609: INFO: metrics-server-954b56d74-7hmq4 started at 2022-11-09 01:05:30 +0000 UTC (0+1 container statuses recorded)
Nov  9 02:42:56.609: INFO: 	Container metrics-server ready: true, restart count 0
Nov  9 02:42:56.609: INFO: etcd-capz-conf-tvyqls-control-plane-7vbm8 started at 2022-11-09 01:04:42 +0000 UTC (0+1 container statuses recorded)
Nov  9 02:42:56.609: INFO: 	Container etcd ready: true, restart count 0
Nov  9 02:42:56.609: INFO: kube-apiserver-capz-conf-tvyqls-control-plane-7vbm8 started at 2022-11-09 01:04:42 +0000 UTC (0+1 container statuses recorded)
Nov  9 02:42:56.609: INFO: 	Container kube-apiserver ready: true, restart count 0
Nov  9 02:42:56.609: INFO: kube-controller-manager-capz-conf-tvyqls-control-plane-7vbm8 started at 2022-11-09 01:04:42 +0000 UTC (0+1 container statuses recorded)
Nov  9 02:42:56.609: INFO: 	Container kube-controller-manager ready: true, restart count 0
Nov  9 02:42:56.609: INFO: kube-scheduler-capz-conf-tvyqls-control-plane-7vbm8 started at 2022-11-09 01:04:42 +0000 UTC (0+1 container statuses recorded)
Nov  9 02:42:56.609: INFO: 	Container kube-scheduler ready: true, restart count 0
Nov  9 02:42:56.609: INFO: kube-proxy-9ns2c started at 2022-11-09 01:04:42 +0000 UTC (0+1 container statuses recorded)
Nov  9 02:42:56.609: INFO: 	Container kube-proxy ready: true, restart count 0
Nov  9 02:42:56.609: INFO: calico-kube-controllers-56c5ff4bf8-gbkvc started at 2022-11-09 01:05:22 +0000 UTC (0+1 container statuses recorded)
Nov  9 02:42:56.609: INFO: 	Container calico-kube-controllers ready: true, restart count 0
Nov  9 02:42:56.609: INFO: coredns-fdff55fb9-4wwnr started at 2022-11-09 01:10:12 +0000 UTC (0+1 container statuses recorded)
Nov  9 02:42:56.609: INFO: 	Container coredns ready: true, restart count 0
Nov  9 02:42:56.786: INFO: 
Latency metrics for node capz-conf-tvyqls-control-plane-7vbm8
[DeferCleanup (Each)] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: CPU)
  tear down framework | framework.go:193
STEP: Destroying namespace "horizontal-pod-autoscaling-1228" for this suite. 11/09/22 02:42:56.787

				Click to see stdout/stderrfrom junit_01.xml

Filter through log files | View test history on testgrid

Kubernetes e2e suite [It] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: CPU) [Serial] [Slow] ReplicationController Should scale from 1 pod to 3 pods and then from 3 pods to 5 pods and verify decision stability 15m41s

go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\s\[sig\-autoscaling\]\s\[Feature\:HPA\]\sHorizontal\spod\sautoscaling\s\(scale\sresource\:\sCPU\)\s\[Serial\]\s\[Slow\]\sReplicationController\sShould\sscale\sfrom\s1\spod\sto\s3\spods\sand\sthen\sfrom\s3\spods\sto\s5\spods\sand\sverify\sdecision\sstability$'

test/e2e/autoscaling/horizontal_pod_autoscaling.go:209
k8s.io/kubernetes/test/e2e/autoscaling.(*HPAScaleTest).run(0xc002043e68, {0x74b75d2?, 0xc0015a1c80?}, {{0x0, 0x0}, {0x74b761c, 0x2}, {0x7506e8a, 0x15}}, 0xc00082cd20)
	test/e2e/autoscaling/horizontal_pod_autoscaling.go:209 +0x2d8
k8s.io/kubernetes/test/e2e/autoscaling.scaleUp({0x74b75d2?, 0x61dc905?}, {{0x0, 0x0}, {0x74b761c, 0x2}, {0x7506e8a, 0x15}}, {0x74b8538, 0x3}, ...)
	test/e2e/autoscaling/horizontal_pod_autoscaling.go:249 +0x212
k8s.io/kubernetes/test/e2e/autoscaling.glob..func6.4.1()
	test/e2e/autoscaling/horizontal_pod_autoscaling.go:81 +0x8b
				[BeforeEach] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: CPU)
  set up framework | framework.go:178
STEP: Creating a kubernetes client 11/09/22 02:07:24.245
Nov  9 02:07:24.245: INFO: >>> kubeConfig: /home/prow/go/src/k8s.io/windows-testing/capz-conf-tvyqls.kubeconfig
I1109 02:07:24.247313   14108 discovery.go:214] Invalidating discovery information
STEP: Building a namespace api object, basename horizontal-pod-autoscaling 11/09/22 02:07:24.247
I1109 02:07:24.284614   14108 reflector.go:221] Starting reflector *v1.ServiceAccount (0s) from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
I1109 02:07:24.284646   14108 reflector.go:257] Listing and watching *v1.ServiceAccount from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
I1109 02:07:24.346214   14108 reflector.go:227] Stopping reflector *v1.ServiceAccount (0s) from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
STEP: Waiting for a default service account to be provisioned in namespace 11/09/22 02:07:24.346
I1109 02:07:24.346605   14108 reflector.go:221] Starting reflector *v1.ServiceAccount (0s) from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
I1109 02:07:24.346619   14108 reflector.go:257] Listing and watching *v1.ServiceAccount from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
I1109 02:07:24.410690   14108 reflector.go:227] Stopping reflector *v1.ServiceAccount (0s) from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
STEP: Waiting for kube-root-ca.crt to be provisioned in namespace 11/09/22 02:07:24.41
I1109 02:07:24.411044   14108 reflector.go:221] Starting reflector *v1.ConfigMap (0s) from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
I1109 02:07:24.411058   14108 reflector.go:257] Listing and watching *v1.ConfigMap from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
I1109 02:07:24.472170   14108 reflector.go:227] Stopping reflector *v1.ConfigMap (0s) from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
[BeforeEach] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: CPU)
  test/e2e/framework/metrics/init/init.go:31
[It] Should scale from 1 pod to 3 pods and then from 3 pods to 5 pods and verify decision stability
  test/e2e/autoscaling/horizontal_pod_autoscaling.go:80
Nov  9 02:07:24.472: INFO: >>> kubeConfig: /home/prow/go/src/k8s.io/windows-testing/capz-conf-tvyqls.kubeconfig
STEP: Running consuming RC rc via /v1, Kind=ReplicationController with 1 replicas 11/09/22 02:07:24.473
STEP: creating replication controller rc in namespace horizontal-pod-autoscaling-3502 11/09/22 02:07:24.526
I1109 02:07:24.563043   14108 runners.go:193] Created replication controller with name: rc, namespace: horizontal-pod-autoscaling-3502, replica count: 1
I1109 02:07:24.563151   14108 reflector.go:221] Starting reflector *v1.Pod (0s) from test/utils/pod_store.go:57
I1109 02:07:24.563161   14108 reflector.go:257] Listing and watching *v1.Pod from test/utils/pod_store.go:57
I1109 02:07:34.614026   14108 runners.go:193] rc Pods: 1 out of 1 created, 1 running, 0 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
STEP: Running controller 11/09/22 02:07:34.614
I1109 02:07:34.614508   14108 reflector.go:227] Stopping reflector *v1.Pod (0s) from test/utils/pod_store.go:57
STEP: creating replication controller rc-ctrl in namespace horizontal-pod-autoscaling-3502 11/09/22 02:07:34.66
I1109 02:07:34.696627   14108 runners.go:193] Created replication controller with name: rc-ctrl, namespace: horizontal-pod-autoscaling-3502, replica count: 1
I1109 02:07:34.696756   14108 reflector.go:221] Starting reflector *v1.Pod (0s) from test/utils/pod_store.go:57
I1109 02:07:34.696782   14108 reflector.go:257] Listing and watching *v1.Pod from test/utils/pod_store.go:57
I1109 02:07:44.747689   14108 runners.go:193] rc-ctrl Pods: 1 out of 1 created, 1 running, 0 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1109 02:07:44.748028   14108 reflector.go:227] Stopping reflector *v1.Pod (0s) from test/utils/pod_store.go:57
Nov  9 02:07:49.749: INFO: Waiting for amount of service:rc-ctrl endpoints to be 1
Nov  9 02:07:49.781: INFO: RC rc: consume 250 millicores in total
Nov  9 02:07:49.781: INFO: RC rc: consume 0 MB in total
Nov  9 02:07:49.781: INFO: RC rc: disabling mem consumption
Nov  9 02:07:49.781: INFO: RC rc: consume custom metric 0 in total
Nov  9 02:07:49.781: INFO: RC rc: disabling consumption of custom metric QPS
Nov  9 02:07:49.782: INFO: RC rc: setting consumption to 250 millicores in total
Nov  9 02:07:49.850: INFO: waiting for 3 replicas (current: 1)
Nov  9 02:08:09.883: INFO: waiting for 3 replicas (current: 1)
Nov  9 02:08:19.785: INFO: RC rc: sending request to consume 250 millicores
Nov  9 02:08:19.785: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-3502/services/rc-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:08:29.883: INFO: waiting for 3 replicas (current: 1)
Nov  9 02:08:49.848: INFO: RC rc: sending request to consume 250 millicores
Nov  9 02:08:49.848: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-3502/services/rc-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:08:49.882: INFO: waiting for 3 replicas (current: 1)
Nov  9 02:09:09.884: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:09:19.892: INFO: RC rc: sending request to consume 250 millicores
Nov  9 02:09:19.893: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-3502/services/rc-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:09:29.883: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:09:49.885: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:09:49.938: INFO: RC rc: sending request to consume 250 millicores
Nov  9 02:09:49.938: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-3502/services/rc-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:10:09.888: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:10:19.979: INFO: RC rc: sending request to consume 250 millicores
Nov  9 02:10:19.979: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-3502/services/rc-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:10:29.887: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:10:49.883: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:10:50.026: INFO: RC rc: sending request to consume 250 millicores
Nov  9 02:10:50.026: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-3502/services/rc-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:11:09.887: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:11:20.070: INFO: RC rc: sending request to consume 250 millicores
Nov  9 02:11:20.070: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-3502/services/rc-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:11:29.885: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:11:49.883: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:11:50.115: INFO: RC rc: sending request to consume 250 millicores
Nov  9 02:11:50.115: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-3502/services/rc-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:12:09.886: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:12:20.159: INFO: RC rc: sending request to consume 250 millicores
Nov  9 02:12:20.159: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-3502/services/rc-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:12:29.883: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:12:49.882: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:12:50.206: INFO: RC rc: sending request to consume 250 millicores
Nov  9 02:12:50.207: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-3502/services/rc-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:13:09.886: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:13:20.251: INFO: RC rc: sending request to consume 250 millicores
Nov  9 02:13:20.251: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-3502/services/rc-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:13:29.887: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:13:49.884: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:13:50.297: INFO: RC rc: sending request to consume 250 millicores
Nov  9 02:13:50.297: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-3502/services/rc-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:14:09.883: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:14:20.342: INFO: RC rc: sending request to consume 250 millicores
Nov  9 02:14:20.342: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-3502/services/rc-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:14:29.888: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:14:49.883: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:14:50.387: INFO: RC rc: sending request to consume 250 millicores
Nov  9 02:14:50.388: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-3502/services/rc-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:15:09.885: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:15:20.435: INFO: RC rc: sending request to consume 250 millicores
Nov  9 02:15:20.435: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-3502/services/rc-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:15:29.884: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:15:49.884: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:15:50.481: INFO: RC rc: sending request to consume 250 millicores
Nov  9 02:15:50.481: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-3502/services/rc-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:16:09.887: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:16:20.528: INFO: RC rc: sending request to consume 250 millicores
Nov  9 02:16:20.528: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-3502/services/rc-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:16:29.888: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:16:49.883: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:16:50.576: INFO: RC rc: sending request to consume 250 millicores
Nov  9 02:16:50.576: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-3502/services/rc-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:17:09.886: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:17:20.621: INFO: RC rc: sending request to consume 250 millicores
Nov  9 02:17:20.621: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-3502/services/rc-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:17:29.883: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:17:49.883: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:17:50.667: INFO: RC rc: sending request to consume 250 millicores
Nov  9 02:17:50.667: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-3502/services/rc-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:18:09.887: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:18:20.715: INFO: RC rc: sending request to consume 250 millicores
Nov  9 02:18:20.715: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-3502/services/rc-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:18:29.883: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:18:49.883: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:18:50.758: INFO: RC rc: sending request to consume 250 millicores
Nov  9 02:18:50.759: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-3502/services/rc-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:19:09.883: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:19:20.806: INFO: RC rc: sending request to consume 250 millicores
Nov  9 02:19:20.806: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-3502/services/rc-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:19:29.883: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:19:49.883: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:19:50.849: INFO: RC rc: sending request to consume 250 millicores
Nov  9 02:19:50.849: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-3502/services/rc-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:20:09.887: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:20:20.895: INFO: RC rc: sending request to consume 250 millicores
Nov  9 02:20:20.895: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-3502/services/rc-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:20:29.887: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:20:49.884: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:20:50.939: INFO: RC rc: sending request to consume 250 millicores
Nov  9 02:20:50.939: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-3502/services/rc-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:21:09.886: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:21:20.985: INFO: RC rc: sending request to consume 250 millicores
Nov  9 02:21:20.985: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-3502/services/rc-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:21:29.888: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:21:49.887: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:21:51.030: INFO: RC rc: sending request to consume 250 millicores
Nov  9 02:21:51.030: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-3502/services/rc-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:22:09.884: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:22:21.075: INFO: RC rc: sending request to consume 250 millicores
Nov  9 02:22:21.075: INFO: ConsumeCPU URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-3502/services/rc-ctrl/proxy/ConsumeCPU  false false durationSec=30&millicores=250&requestSizeMillicores=100  }
Nov  9 02:22:29.883: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:22:49.883: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:22:49.915: INFO: waiting for 3 replicas (current: 2)
Nov  9 02:22:49.916: INFO: Unexpected error: timeout waiting 15m0s for 3 replicas: 
    <*errors.errorString | 0xc000287c80>: {
        s: "timed out waiting for the condition",
    }
Nov  9 02:22:49.916: FAIL: timeout waiting 15m0s for 3 replicas: timed out waiting for the condition

Full Stack Trace
k8s.io/kubernetes/test/e2e/autoscaling.(*HPAScaleTest).run(0xc002043e68, {0x74b75d2?, 0xc0015a1c80?}, {{0x0, 0x0}, {0x74b761c, 0x2}, {0x7506e8a, 0x15}}, 0xc00082cd20)
	test/e2e/autoscaling/horizontal_pod_autoscaling.go:209 +0x2d8
k8s.io/kubernetes/test/e2e/autoscaling.scaleUp({0x74b75d2?, 0x61dc905?}, {{0x0, 0x0}, {0x74b761c, 0x2}, {0x7506e8a, 0x15}}, {0x74b8538, 0x3}, ...)
	test/e2e/autoscaling/horizontal_pod_autoscaling.go:249 +0x212
k8s.io/kubernetes/test/e2e/autoscaling.glob..func6.4.1()
	test/e2e/autoscaling/horizontal_pod_autoscaling.go:81 +0x8b
STEP: Removing consuming RC rc 11/09/22 02:22:49.955
Nov  9 02:22:49.955: INFO: RC rc: stopping metric consumer
Nov  9 02:22:49.955: INFO: RC rc: stopping CPU consumer
Nov  9 02:22:49.955: INFO: RC rc: stopping mem consumer
STEP: deleting ReplicationController rc in namespace horizontal-pod-autoscaling-3502, will wait for the garbage collector to delete the pods 11/09/22 02:22:59.959
I1109 02:22:59.993903   14108 reflector.go:221] Starting reflector *v1.Pod (0s) from test/utils/pod_store.go:57
I1109 02:22:59.993953   14108 reflector.go:257] Listing and watching *v1.Pod from test/utils/pod_store.go:57
Nov  9 02:23:00.079: INFO: Deleting ReplicationController rc took: 35.575488ms
I1109 02:23:00.180467   14108 resource.go:655] Ignoring inactive pod horizontal-pod-autoscaling-3502/rc-j6sgm in state Running, deletion time 2022-11-09 02:23:01 +0000 UTC
I1109 02:23:00.180526   14108 resource.go:655] Ignoring inactive pod horizontal-pod-autoscaling-3502/rc-jmlkk in state Running, deletion time 2022-11-09 02:23:01 +0000 UTC
Nov  9 02:23:00.180: INFO: Terminating ReplicationController rc pods took: 100.901437ms
I1109 02:23:02.281914   14108 reflector.go:227] Stopping reflector *v1.Pod (0s) from test/utils/pod_store.go:57
STEP: deleting ReplicationController rc-ctrl in namespace horizontal-pod-autoscaling-3502, will wait for the garbage collector to delete the pods 11/09/22 02:23:02.349
I1109 02:23:02.381492   14108 reflector.go:221] Starting reflector *v1.Pod (0s) from test/utils/pod_store.go:57
I1109 02:23:02.381536   14108 reflector.go:257] Listing and watching *v1.Pod from test/utils/pod_store.go:57
Nov  9 02:23:02.467: INFO: Deleting ReplicationController rc-ctrl took: 35.448951ms
I1109 02:23:02.568141   14108 resource.go:655] Ignoring inactive pod horizontal-pod-autoscaling-3502/rc-ctrl-zv2jj in state Running, deletion time 2022-11-09 02:23:03 +0000 UTC
Nov  9 02:23:02.568: INFO: Terminating ReplicationController rc-ctrl pods took: 100.979048ms
I1109 02:23:04.569392   14108 reflector.go:227] Stopping reflector *v1.Pod (0s) from test/utils/pod_store.go:57
[AfterEach] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: CPU)
  test/e2e/framework/node/init/init.go:32
Nov  9 02:23:04.626: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
[DeferCleanup (Each)] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: CPU)
  test/e2e/framework/metrics/init/init.go:33
[DeferCleanup (Each)] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: CPU)
  dump namespaces | framework.go:196
STEP: dump namespace information after failure 11/09/22 02:23:04.672
STEP: Collecting events from namespace "horizontal-pod-autoscaling-3502". 11/09/22 02:23:04.672
STEP: Found 19 events. 11/09/22 02:23:04.71
Nov  9 02:23:04.710: INFO: At 2022-11-09 02:07:24 +0000 UTC - event for rc: {replication-controller } SuccessfulCreate: Created pod: rc-j6sgm
Nov  9 02:23:04.710: INFO: At 2022-11-09 02:07:24 +0000 UTC - event for rc-j6sgm: {default-scheduler } Scheduled: Successfully assigned horizontal-pod-autoscaling-3502/rc-j6sgm to capz-conf-hhzhh
Nov  9 02:23:04.710: INFO: At 2022-11-09 02:07:28 +0000 UTC - event for rc-j6sgm: {kubelet capz-conf-hhzhh} Created: Created container rc
Nov  9 02:23:04.710: INFO: At 2022-11-09 02:07:28 +0000 UTC - event for rc-j6sgm: {kubelet capz-conf-hhzhh} Pulled: Container image "registry.k8s.io/e2e-test-images/resource-consumer:1.13" already present on machine
Nov  9 02:23:04.710: INFO: At 2022-11-09 02:07:31 +0000 UTC - event for rc-j6sgm: {kubelet capz-conf-hhzhh} Started: Started container rc
Nov  9 02:23:04.710: INFO: At 2022-11-09 02:07:34 +0000 UTC - event for rc-ctrl: {replication-controller } SuccessfulCreate: Created pod: rc-ctrl-zv2jj
Nov  9 02:23:04.710: INFO: At 2022-11-09 02:07:34 +0000 UTC - event for rc-ctrl-zv2jj: {default-scheduler } Scheduled: Successfully assigned horizontal-pod-autoscaling-3502/rc-ctrl-zv2jj to capz-conf-m97cx
Nov  9 02:23:04.710: INFO: At 2022-11-09 02:07:38 +0000 UTC - event for rc-ctrl-zv2jj: {kubelet capz-conf-m97cx} Created: Created container rc-ctrl
Nov  9 02:23:04.710: INFO: At 2022-11-09 02:07:38 +0000 UTC - event for rc-ctrl-zv2jj: {kubelet capz-conf-m97cx} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.40" already present on machine
Nov  9 02:23:04.710: INFO: At 2022-11-09 02:07:41 +0000 UTC - event for rc-ctrl-zv2jj: {kubelet capz-conf-m97cx} Started: Started container rc-ctrl
Nov  9 02:23:04.710: INFO: At 2022-11-09 02:08:49 +0000 UTC - event for rc: {replication-controller } SuccessfulCreate: Created pod: rc-jmlkk
Nov  9 02:23:04.710: INFO: At 2022-11-09 02:08:49 +0000 UTC - event for rc: {horizontal-pod-autoscaler } SuccessfulRescale: New size: 2; reason: cpu resource utilization (percentage of request) above target
Nov  9 02:23:04.710: INFO: At 2022-11-09 02:08:49 +0000 UTC - event for rc-jmlkk: {default-scheduler } Scheduled: Successfully assigned horizontal-pod-autoscaling-3502/rc-jmlkk to capz-conf-m97cx
Nov  9 02:23:04.710: INFO: At 2022-11-09 02:08:53 +0000 UTC - event for rc-jmlkk: {kubelet capz-conf-m97cx} Pulled: Container image "registry.k8s.io/e2e-test-images/resource-consumer:1.13" already present on machine
Nov  9 02:23:04.710: INFO: At 2022-11-09 02:08:53 +0000 UTC - event for rc-jmlkk: {kubelet capz-conf-m97cx} Created: Created container rc
Nov  9 02:23:04.710: INFO: At 2022-11-09 02:08:56 +0000 UTC - event for rc-jmlkk: {kubelet capz-conf-m97cx} Started: Started container rc
Nov  9 02:23:04.710: INFO: At 2022-11-09 02:23:00 +0000 UTC - event for rc-j6sgm: {kubelet capz-conf-hhzhh} Killing: Stopping container rc
Nov  9 02:23:04.710: INFO: At 2022-11-09 02:23:00 +0000 UTC - event for rc-jmlkk: {kubelet capz-conf-m97cx} Killing: Stopping container rc
Nov  9 02:23:04.710: INFO: At 2022-11-09 02:23:02 +0000 UTC - event for rc-ctrl-zv2jj: {kubelet capz-conf-m97cx} Killing: Stopping container rc-ctrl
Nov  9 02:23:04.741: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
Nov  9 02:23:04.741: INFO: 
Nov  9 02:23:04.786: INFO: 
Logging node info for node capz-conf-hhzhh
Nov  9 02:23:04.822: INFO: Node Info: &Node{ObjectMeta:{capz-conf-hhzhh    641b6838-eeea-48b0-9dc0-0ff9ab096f10 10430 0 2022-11-09 01:07:16 +0000 UTC <nil> <nil> map[agentpool:windowsgmsa beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:windows failure-domain.beta.kubernetes.io/region:eastus failure-domain.beta.kubernetes.io/zone:0 kubernetes.io/arch:amd64 kubernetes.io/hostname:capz-conf-hhzhh kubernetes.io/os:windows node.kubernetes.io/instance-type:Standard_D4s_v3 node.kubernetes.io/windows-build:10.0.20348 topology.kubernetes.io/region:eastus topology.kubernetes.io/zone:0] map[cluster.x-k8s.io/cluster-name:capz-conf-tvyqls cluster.x-k8s.io/cluster-namespace:default cluster.x-k8s.io/machine:capz-conf-tvyqls-md-win-7848cfdb84-9mln2 cluster.x-k8s.io/owner-kind:MachineSet cluster.x-k8s.io/owner-name:capz-conf-tvyqls-md-win-7848cfdb84 kubeadm.alpha.kubernetes.io/cri-socket:npipe:////./pipe/containerd-containerd node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:10.1.0.4/16 projectcalico.org/IPv4VXLANTunnelAddr:192.168.84.65 projectcalico.org/VXLANTunnelMACAddr:00:15:5d:5b:b6:ce volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubeadm Update v1 2022-11-09 01:07:16 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:kubeadm.alpha.kubernetes.io/cri-socket":{}}}} } {kubelet.exe Update v1 2022-11-09 01:07:16 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:node.kubernetes.io/windows-build":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {manager Update v1 2022-11-09 01:08:02 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:cluster.x-k8s.io/cluster-name":{},"f:cluster.x-k8s.io/cluster-namespace":{},"f:cluster.x-k8s.io/machine":{},"f:cluster.x-k8s.io/owner-kind":{},"f:cluster.x-k8s.io/owner-name":{}}}} } {kube-controller-manager Update v1 2022-11-09 01:08:06 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}}} } {Go-http-client Update v1 2022-11-09 01:08:32 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4VXLANTunnelAddr":{},"f:projectcalico.org/VXLANTunnelMACAddr":{}}}} status} {cluster-api-e2e Update v1 2022-11-09 01:10:08 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:agentpool":{}}}} } {e2e.test Update v1 2022-11-09 01:22:08 +0000 UTC FieldsV1 {"f:status":{"f:capacity":{"f:example.com/fakePTSRes":{}}}} status} {kubelet.exe Update v1 2022-11-09 02:18:23 +0000 UTC FieldsV1 {"f:status":{"f:allocatable":{"f:example.com/fakePTSRes":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:,DoNotUseExternalID:,ProviderID:azure:///subscriptions/===REDACTED===/resourceGroups/capz-conf-tvyqls/providers/Microsoft.Compute/virtualMachines/capz-conf-hhzhh,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-azure-disk: {{8 0} {<nil>} 8 DecimalSI},cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{136912564224 0} {<nil>} 133703676Ki BinarySI},example.com/fakePTSRes: {{10 0} {<nil>} 10 DecimalSI},memory: {{17179398144 0} {<nil>} 16776756Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-azure-disk: {{8 0} {<nil>} 8 DecimalSI},cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{123221307598 0} {<nil>} 123221307598 DecimalSI},example.com/fakePTSRes: {{10 0} {<nil>} 10 DecimalSI},memory: {{17074540544 0} {<nil>} 16674356Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-09 02:18:23 +0000 UTC,LastTransitionTime:2022-11-09 01:07:16 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-09 02:18:23 +0000 UTC,LastTransitionTime:2022-11-09 01:07:16 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-09 02:18:23 +0000 UTC,LastTransitionTime:2022-11-09 01:07:16 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-09 02:18:23 +0000 UTC,LastTransitionTime:2022-11-09 01:08:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status,},},Addresses:[]NodeAddress{NodeAddress{Type:Hostname,Address:capz-conf-hhzhh,},NodeAddress{Type:InternalIP,Address:10.1.0.4,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:capz-conf-hhzhh,SystemUUID:5F9EA3E7-22F1-406E-9AAC-69BA5DD83BF4,BootID:8,KernelVersion:10.0.20348.1129,OSImage:Windows Server 2022 Datacenter,ContainerRuntimeVersion:containerd://1.6.8,KubeletVersion:v1.26.0-alpha.3.425+2f2240400391ad,KubeProxyVersion:v1.26.0-alpha.3.425+2f2240400391ad,OperatingSystem:windows,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:307641217,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:221349084,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:218942479,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:183620591,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:181784786,},ContainerImage{Names:[ghcr.io/kubernetes-sigs/sig-windows/eventflow-logger@sha256:63bf2aa9db909d0d90fb5205abf7fb2a6d9a494b89cbd2508a42457dfc875505 ghcr.io/kubernetes-sigs/sig-windows/eventflow-logger:v0.1.0],SizeBytes:148054279,},ContainerImage{Names:[ghcr.io/kubernetes-sigs/sig-windows/csi-proxy@sha256:96b4144986319a747ba599892454be2737aae6005d96b8e13ed481321ac3afba ghcr.io/kubernetes-sigs/sig-windows/csi-proxy:v1.0.2],SizeBytes:123925937,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/resource-consumer@sha256:ba5e047a337e5d0709bc57df45b95b2c7f6f2794b290e4e24f7fc8980d60b25a registry.k8s.io/e2e-test-images/resource-consumer:1.13],SizeBytes:121182992,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6],SizeBytes:119397646,},ContainerImage{Names:[docker.io/sigwindowstools/kube-proxy:v1.23.1-calico-hostprocess docker.io/sigwindowstools/kube-proxy:v1.26.0-alpha.3.425_2f2240400391ad-calico-hostprocess],SizeBytes:116182072,},ContainerImage{Names:[docker.io/sigwindowstools/calico-install@sha256:97bc10aa5000a0ee1c842ac32771fe7a45a3a5ca507711bdf57ae2eb5f293e2b docker.io/sigwindowstools/calico-install:v3.23.0-hostprocess],SizeBytes:47258343,},ContainerImage{Names:[docker.io/sigwindowstools/calico-node@sha256:ea8b55bde9aed6a649582a6e21029577430661c743d94b3a5e93d57e648874a2 docker.io/sigwindowstools/calico-node:v3.23.0-hostprocess],SizeBytes:27005624,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Nov  9 02:23:04.822: INFO: 
Logging kubelet events for node capz-conf-hhzhh
Nov  9 02:23:04.854: INFO: 
Logging pods the kubelet thinks is on node capz-conf-hhzhh
Nov  9 02:23:04.939: INFO: kube-proxy-windows-ldnh5 started at 2022-11-09 01:07:16 +0000 UTC (0+1 container statuses recorded)
Nov  9 02:23:04.939: INFO: 	Container kube-proxy ready: true, restart count 0
Nov  9 02:23:04.939: INFO: calico-node-windows-jpm48 started at 2022-11-09 01:07:16 +0000 UTC (1+2 container statuses recorded)
Nov  9 02:23:04.939: INFO: 	Init container install-cni ready: true, restart count 0
Nov  9 02:23:04.939: INFO: 	Container calico-node-felix ready: true, restart count 1
Nov  9 02:23:04.939: INFO: 	Container calico-node-startup ready: true, restart count 0
Nov  9 02:23:04.939: INFO: containerd-logger-z56l7 started at 2022-11-09 01:07:16 +0000 UTC (0+1 container statuses recorded)
Nov  9 02:23:04.939: INFO: 	Container containerd-logger ready: true, restart count 0
Nov  9 02:23:04.939: INFO: csi-proxy-dkn8s started at 2022-11-09 01:08:06 +0000 UTC (0+1 container statuses recorded)
Nov  9 02:23:04.939: INFO: 	Container csi-proxy ready: true, restart count 0
Nov  9 02:23:05.150: INFO: 
Latency metrics for node capz-conf-hhzhh
Nov  9 02:23:05.150: INFO: 
Logging node info for node capz-conf-m97cx
Nov  9 02:23:05.183: INFO: Node Info: &Node{ObjectMeta:{capz-conf-m97cx    36965107-96dc-426b-924d-c291c94a2d04 10417 0 2022-11-09 01:07:00 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:windows failure-domain.beta.kubernetes.io/region:eastus failure-domain.beta.kubernetes.io/zone:0 kubernetes.io/arch:amd64 kubernetes.io/hostname:capz-conf-m97cx kubernetes.io/os:windows node.kubernetes.io/instance-type:Standard_D4s_v3 node.kubernetes.io/windows-build:10.0.20348 topology.kubernetes.io/region:eastus topology.kubernetes.io/zone:0] map[cluster.x-k8s.io/cluster-name:capz-conf-tvyqls cluster.x-k8s.io/cluster-namespace:default cluster.x-k8s.io/machine:capz-conf-tvyqls-md-win-7848cfdb84-cw8bk cluster.x-k8s.io/owner-kind:MachineSet cluster.x-k8s.io/owner-name:capz-conf-tvyqls-md-win-7848cfdb84 kubeadm.alpha.kubernetes.io/cri-socket:npipe:////./pipe/containerd-containerd node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:10.1.0.5/16 projectcalico.org/IPv4VXLANTunnelAddr:192.168.100.1 projectcalico.org/VXLANTunnelMACAddr:00:15:5d:ac:92:68 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubeadm Update v1 2022-11-09 01:07:00 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:kubeadm.alpha.kubernetes.io/cri-socket":{}}}} } {kubelet.exe Update v1 2022-11-09 01:07:00 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:node.kubernetes.io/windows-build":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-11-09 01:07:31 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}}} } {manager Update v1 2022-11-09 01:08:02 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:cluster.x-k8s.io/cluster-name":{},"f:cluster.x-k8s.io/cluster-namespace":{},"f:cluster.x-k8s.io/machine":{},"f:cluster.x-k8s.io/owner-kind":{},"f:cluster.x-k8s.io/owner-name":{}}}} } {Go-http-client Update v1 2022-11-09 01:08:08 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4VXLANTunnelAddr":{},"f:projectcalico.org/VXLANTunnelMACAddr":{}}}} status} {e2e.test Update v1 2022-11-09 01:22:07 +0000 UTC FieldsV1 {"f:status":{"f:capacity":{"f:example.com/fakePTSRes":{}}}} status} {kubelet.exe Update v1 2022-11-09 02:18:16 +0000 UTC FieldsV1 {"f:status":{"f:allocatable":{"f:example.com/fakePTSRes":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:,DoNotUseExternalID:,ProviderID:azure:///subscriptions/===REDACTED===/resourceGroups/capz-conf-tvyqls/providers/Microsoft.Compute/virtualMachines/capz-conf-m97cx,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-azure-disk: {{8 0} {<nil>} 8 DecimalSI},cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{136912564224 0} {<nil>} 133703676Ki BinarySI},example.com/fakePTSRes: {{10 0} {<nil>} 10 DecimalSI},memory: {{17179398144 0} {<nil>} 16776756Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-azure-disk: {{8 0} {<nil>} 8 DecimalSI},cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{123221307598 0} {<nil>} 123221307598 DecimalSI},example.com/fakePTSRes: {{10 0} {<nil>} 10 DecimalSI},memory: {{17074540544 0} {<nil>} 16674356Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-09 02:18:16 +0000 UTC,LastTransitionTime:2022-11-09 01:07:00 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-09 02:18:16 +0000 UTC,LastTransitionTime:2022-11-09 01:07:00 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-09 02:18:16 +0000 UTC,LastTransitionTime:2022-11-09 01:07:00 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-09 02:18:16 +0000 UTC,LastTransitionTime:2022-11-09 01:07:31 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status,},},Addresses:[]NodeAddress{NodeAddress{Type:Hostname,Address:capz-conf-m97cx,},NodeAddress{Type:InternalIP,Address:10.1.0.5,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:capz-conf-m97cx,SystemUUID:FC504DDA-1329-4BBA-8C1E-77FAD783BCA1,BootID:8,KernelVersion:10.0.20348.1129,OSImage:Windows Server 2022 Datacenter,ContainerRuntimeVersion:containerd://1.6.8,KubeletVersion:v1.26.0-alpha.3.425+2f2240400391ad,KubeProxyVersion:v1.26.0-alpha.3.425+2f2240400391ad,OperatingSystem:windows,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:307641217,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:221349084,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:218942479,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:183620591,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:181784786,},ContainerImage{Names:[ghcr.io/kubernetes-sigs/sig-windows/eventflow-logger@sha256:63bf2aa9db909d0d90fb5205abf7fb2a6d9a494b89cbd2508a42457dfc875505 ghcr.io/kubernetes-sigs/sig-windows/eventflow-logger:v0.1.0],SizeBytes:148054279,},ContainerImage{Names:[ghcr.io/kubernetes-sigs/sig-windows/csi-proxy@sha256:96b4144986319a747ba599892454be2737aae6005d96b8e13ed481321ac3afba ghcr.io/kubernetes-sigs/sig-windows/csi-proxy:v1.0.2],SizeBytes:123925937,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/resource-consumer@sha256:ba5e047a337e5d0709bc57df45b95b2c7f6f2794b290e4e24f7fc8980d60b25a registry.k8s.io/e2e-test-images/resource-consumer:1.13],SizeBytes:121182992,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6],SizeBytes:119397646,},ContainerImage{Names:[docker.io/sigwindowstools/kube-proxy:v1.23.1-calico-hostprocess docker.io/sigwindowstools/kube-proxy:v1.26.0-alpha.3.425_2f2240400391ad-calico-hostprocess],SizeBytes:116182072,},ContainerImage{Names:[docker.io/sigwindowstools/calico-install@sha256:97bc10aa5000a0ee1c842ac32771fe7a45a3a5ca507711bdf57ae2eb5f293e2b docker.io/sigwindowstools/calico-install:v3.23.0-hostprocess],SizeBytes:47258343,},ContainerImage{Names:[docker.io/sigwindowstools/calico-node@sha256:ea8b55bde9aed6a649582a6e21029577430661c743d94b3a5e93d57e648874a2 docker.io/sigwindowstools/calico-node:v3.23.0-hostprocess],SizeBytes:27005624,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Nov  9 02:23:05.183: INFO: 
Logging kubelet events for node capz-conf-m97cx
Nov  9 02:23:05.215: INFO: 
Logging pods the kubelet thinks is on node capz-conf-m97cx
Nov  9 02:23:05.269: INFO: calico-node-windows-w5q9w started at 2022-11-09 01:07:01 +0000 UTC (1+2 container statuses recorded)
Nov  9 02:23:05.269: INFO: 	Init container install-cni ready: true, restart count 0
Nov  9 02:23:05.269: INFO: 	Container calico-node-felix ready: true, restart count 1
Nov  9 02:23:05.269: INFO: 	Container calico-node-startup ready: true, restart count 0
Nov  9 02:23:05.269: INFO: containerd-logger-v9qh5 started at 2022-11-09 01:07:01 +0000 UTC (0+1 container statuses recorded)
Nov  9 02:23:05.269: INFO: 	Container containerd-logger ready: true, restart count 0
Nov  9 02:23:05.269: INFO: kube-proxy-windows-55jdx started at 2022-11-09 01:07:01 +0000 UTC (0+1 container statuses recorded)
Nov  9 02:23:05.269: INFO: 	Container kube-proxy ready: true, restart count 0
Nov  9 02:23:05.269: INFO: csi-proxy-tn9x7 started at 2022-11-09 01:07:31 +0000 UTC (0+1 container statuses recorded)
Nov  9 02:23:05.269: INFO: 	Container csi-proxy ready: true, restart count 0
Nov  9 02:23:05.437: INFO: 
Latency metrics for node capz-conf-m97cx
Nov  9 02:23:05.437: INFO: 
Logging node info for node capz-conf-tvyqls-control-plane-7vbm8
Nov  9 02:23:05.470: INFO: Node Info: &Node{ObjectMeta:{capz-conf-tvyqls-control-plane-7vbm8    59067037-dec8-4e79-8a8a-d150ea7ce6ba 10493 0 2022-11-09 01:04:38 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D2s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eastus failure-domain.beta.kubernetes.io/zone:eastus-3 kubernetes.io/arch:amd64 kubernetes.io/hostname:capz-conf-tvyqls-control-plane-7vbm8 kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:Standard_D2s_v3 topology.kubernetes.io/region:eastus topology.kubernetes.io/zone:eastus-3] map[cluster.x-k8s.io/cluster-name:capz-conf-tvyqls cluster.x-k8s.io/cluster-namespace:default cluster.x-k8s.io/machine:capz-conf-tvyqls-control-plane-ljqrh cluster.x-k8s.io/owner-kind:KubeadmControlPlane cluster.x-k8s.io/owner-name:capz-conf-tvyqls-control-plane kubeadm.alpha.kubernetes.io/cri-socket:unix:///var/run/containerd/containerd.sock node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:10.0.0.4/16 projectcalico.org/IPv4VXLANTunnelAddr:192.168.2.192 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-11-09 01:04:38 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kubeadm Update v1 2022-11-09 01:04:40 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:kubeadm.alpha.kubernetes.io/cri-socket":{}},"f:labels":{"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {manager Update v1 2022-11-09 01:04:55 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:cluster.x-k8s.io/cluster-name":{},"f:cluster.x-k8s.io/cluster-namespace":{},"f:cluster.x-k8s.io/machine":{},"f:cluster.x-k8s.io/owner-kind":{},"f:cluster.x-k8s.io/owner-name":{}}}} } {kube-controller-manager Update v1 2022-11-09 01:05:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:taints":{}}} } {Go-http-client Update v1 2022-11-09 01:05:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4VXLANTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-09 02:19:03 +0000 UTC FieldsV1 {"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:,DoNotUseExternalID:,ProviderID:azure:///subscriptions/===REDACTED===/resourceGroups/capz-conf-tvyqls/providers/Microsoft.Compute/virtualMachines/capz-conf-tvyqls-control-plane-7vbm8,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-azure-disk: {{4 0} {<nil>} 4 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{133003395072 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8344723456 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-azure-disk: {{4 0} {<nil>} 4 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{119703055367 0} {<nil>} 119703055367 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8239865856 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-09 01:05:27 +0000 UTC,LastTransitionTime:2022-11-09 01:05:27 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-09 02:19:03 +0000 UTC,LastTransitionTime:2022-11-09 01:04:17 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-09 02:19:03 +0000 UTC,LastTransitionTime:2022-11-09 01:04:17 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-09 02:19:03 +0000 UTC,LastTransitionTime:2022-11-09 01:04:17 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-09 02:19:03 +0000 UTC,LastTransitionTime:2022-11-09 01:05:22 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:Hostname,Address:capz-conf-tvyqls-control-plane-7vbm8,},NodeAddress{Type:InternalIP,Address:10.0.0.4,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:37a2e1ceaa0d455a8de0dda7f144eb7f,SystemUUID:50325a86-a564-f642-b7cf-2b4ce49e7328,BootID:49adcb05-103e-4672-b80c-d3eea5cf97ca,KernelVersion:5.4.0-1091-azure,OSImage:Ubuntu 18.04.6 LTS,ContainerRuntimeVersion:containerd://1.6.2,KubeletVersion:v1.26.0-alpha.3.425+2f2240400391ad,KubeProxyVersion:v1.26.0-alpha.3.425+2f2240400391ad,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[docker.io/calico/cni@sha256:914823d144204288f881e49b93b6852febfe669074cd4e2a782860981615f521 docker.io/calico/cni:v3.23.0],SizeBytes:110494683,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:b83c1d70989e1fe87583607bf5aee1ee34e52773d4755b95f5cf5a451962f3a4 registry.k8s.io/etcd:3.5.5-0],SizeBytes:102417044,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:102157811,},ContainerImage{Names:[docker.io/calico/node@sha256:4763820ecb4d8e82483a2ffabfec7fcded9603318692df210a778d223a4d7474 docker.io/calico/node:v3.23.0],SizeBytes:71573794,},ContainerImage{Names:[docker.io/calico/kube-controllers@sha256:78bc199299f966b0694dc4044501aee2d7ebd6862b2b0a00bca3ee8d3813c82f docker.io/calico/kube-controllers:v3.23.0],SizeBytes:56343954,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:51155161,},ContainerImage{Names:[gcr.io/k8s-staging-ci-images/kube-apiserver@sha256:f4a034110c7fecfbd53047bb4e7a568b1d123694426129781f718b2b83f650dd gcr.io/k8s-staging-ci-images/kube-apiserver:v1.26.0-alpha.3.425_2f2240400391ad],SizeBytes:34983033,},ContainerImage{Names:[registry.k8s.io/kube-apiserver@sha256:4188262a351f156e8027ff81693d771c35b34b668cbd61e59c4a4490dd5c08f3 registry.k8s.io/kube-apiserver:v1.25.3],SizeBytes:34238163,},ContainerImage{Names:[gcr.io/k8s-staging-ci-images/kube-controller-manager@sha256:4660397406f51cc97599ef65f46644d7cffad440bc3169fdfc0224626b3b2780 gcr.io/k8s-staging-ci-images/kube-controller-manager:v1.26.0-alpha.3.425_2f2240400391ad],SizeBytes:31999085,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager@sha256:d3a06262256f3e7578d5f77df137a8cdf58f9f498f35b5b56d116e8a7e31dc91 registry.k8s.io/kube-controller-manager:v1.25.3],SizeBytes:31261869,},ContainerImage{Names:[k8s.gcr.io/metrics-server/metrics-server@sha256:6385aec64bb97040a5e692947107b81e178555c7a5b71caa90d733e4130efc10 k8s.gcr.io/metrics-server/metrics-server:v0.5.2],SizeBytes:26023008,},ContainerImage{Names:[gcr.io/k8s-staging-ci-images/kube-proxy@sha256:8dcde74eedeada05ac0d0b3fec95ca4299544d64c362e7968f7f5ecb69186030 gcr.io/k8s-staging-ci-images/kube-proxy:v1.26.0-alpha.3.425_2f2240400391ad],SizeBytes:21362412,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:6bf25f038543e1f433cb7f2bdda445ed348c7b9279935ebc2ae4f432308ed82f registry.k8s.io/kube-proxy:v1.25.3],SizeBytes:20265805,},ContainerImage{Names:[gcr.io/k8s-staging-ci-images/kube-scheduler@sha256:fb955dc0dd068e8c525ac60042dac1c9cd349272855639394d761125851f8397 gcr.io/k8s-staging-ci-images/kube-scheduler:v1.26.0-alpha.3.425_2f2240400391ad],SizeBytes:17278982,},ContainerImage{Names:[registry.k8s.io/kube-scheduler@sha256:f478aa916568b00269068ff1e9ff742ecc16192eb6e371e30f69f75df904162e registry.k8s.io/kube-scheduler:v1.25.3],SizeBytes:15798744,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[docker.io/sigwindowstools/k8s-gmsa-webhook@sha256:56bba3b5a4788c05dc45f51444b924f6bed05a29789e04e43a6e2c3dbb07d9ce docker.io/sigwindowstools/k8s-gmsa-webhook:latest],SizeBytes:8874257,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Nov  9 02:23:05.470: INFO: 
Logging kubelet events for node capz-conf-tvyqls-control-plane-7vbm8
Nov  9 02:23:05.502: INFO: 
Logging pods the kubelet thinks is on node capz-conf-tvyqls-control-plane-7vbm8
Nov  9 02:23:05.557: INFO: metrics-server-954b56d74-7hmq4 started at 2022-11-09 01:05:30 +0000 UTC (0+1 container statuses recorded)
Nov  9 02:23:05.557: INFO: 	Container metrics-server ready: true, restart count 0
Nov  9 02:23:05.557: INFO: etcd-capz-conf-tvyqls-control-plane-7vbm8 started at 2022-11-09 01:04:42 +0000 UTC (0+1 container statuses recorded)
Nov  9 02:23:05.557: INFO: 	Container etcd ready: true, restart count 0
Nov  9 02:23:05.557: INFO: kube-apiserver-capz-conf-tvyqls-control-plane-7vbm8 started at 2022-11-09 01:04:42 +0000 UTC (0+1 container statuses recorded)
Nov  9 02:23:05.557: INFO: 	Container kube-apiserver ready: true, restart count 0
Nov  9 02:23:05.557: INFO: kube-controller-manager-capz-conf-tvyqls-control-plane-7vbm8 started at 2022-11-09 01:04:42 +0000 UTC (0+1 container statuses recorded)
Nov  9 02:23:05.557: INFO: 	Container kube-controller-manager ready: true, restart count 0
Nov  9 02:23:05.557: INFO: kube-scheduler-capz-conf-tvyqls-control-plane-7vbm8 started at 2022-11-09 01:04:42 +0000 UTC (0+1 container statuses recorded)
Nov  9 02:23:05.557: INFO: 	Container kube-scheduler ready: true, restart count 0
Nov  9 02:23:05.557: INFO: kube-proxy-9ns2c started at 2022-11-09 01:04:42 +0000 UTC (0+1 container statuses recorded)
Nov  9 02:23:05.557: INFO: 	Container kube-proxy ready: true, restart count 0
Nov  9 02:23:05.557: INFO: calico-kube-controllers-56c5ff4bf8-gbkvc started at 2022-11-09 01:05:22 +0000 UTC (0+1 container statuses recorded)
Nov  9 02:23:05.557: INFO: 	Container calico-kube-controllers ready: true, restart count 0
Nov  9 02:23:05.557: INFO: coredns-fdff55fb9-4wwnr started at 2022-11-09 01:10:12 +0000 UTC (0+1 container statuses recorded)
Nov  9 02:23:05.557: INFO: 	Container coredns ready: true, restart count 0
Nov  9 02:23:05.557: INFO: calico-node-7bphx started at 2022-11-09 01:05:02 +0000 UTC (2+1 container statuses recorded)
Nov  9 02:23:05.557: INFO: 	Init container upgrade-ipam ready: true, restart count 0
Nov  9 02:23:05.557: INFO: 	Init container install-cni ready: true, restart count 0
Nov  9 02:23:05.557: INFO: 	Container calico-node ready: true, restart count 0
Nov  9 02:23:05.557: INFO: coredns-fdff55fb9-vl42q started at 2022-11-09 01:10:12 +0000 UTC (0+1 container statuses recorded)
Nov  9 02:23:05.557: INFO: 	Container coredns ready: true, restart count 0
Nov  9 02:23:05.728: INFO: 
Latency metrics for node capz-conf-tvyqls-control-plane-7vbm8
[DeferCleanup (Each)] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: CPU)
  tear down framework | framework.go:193
STEP: Destroying namespace "horizontal-pod-autoscaling-3502" for this suite. 11/09/22 02:23:05.728

				Click to see stdout/stderrfrom junit_01.xml

Filter through log files | View test history on testgrid

Kubernetes e2e suite [It] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: Memory) [Serial] [Slow] Deployment (Pod Resource) Should scale from 1 pod to 3 pods and then from 3 pods to 5 pods using Average Utilization for aggregation 15m41s

go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\s\[sig\-autoscaling\]\s\[Feature\:HPA\]\sHorizontal\spod\sautoscaling\s\(scale\sresource\:\sMemory\)\s\[Serial\]\s\[Slow\]\sDeployment\s\(Pod\sResource\)\sShould\sscale\sfrom\s1\spod\sto\s3\spods\sand\sthen\sfrom\s3\spods\sto\s5\spods\susing\sAverage\sUtilization\sfor\saggregation$'

test/e2e/autoscaling/horizontal_pod_autoscaling.go:209
k8s.io/kubernetes/test/e2e/autoscaling.(*HPAScaleTest).run(0xc0031dde68, {0x74e3c28?, 0xc001fa4180?}, {{0x74b92e8, 0x4}, {0x74c2479, 0x7}, {0x74ca886, 0xa}}, 0xc00082ce10)
	test/e2e/autoscaling/horizontal_pod_autoscaling.go:209 +0x2d8
k8s.io/kubernetes/test/e2e/autoscaling.scaleUp({0x74e3c28?, 0x61dc905?}, {{0x74b92e8, 0x4}, {0x74c2479, 0x7}, {0x74ca886, 0xa}}, {0x74bdff0, 0x6}, ...)
	test/e2e/autoscaling/horizontal_pod_autoscaling.go:249 +0x212
k8s.io/kubernetes/test/e2e/autoscaling.glob..func7.1.1()
	test/e2e/autoscaling/horizontal_pod_autoscaling.go:155 +0x88
				[BeforeEach] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: Memory)
  set up framework | framework.go:178
STEP: Creating a kubernetes client 11/09/22 05:37:34.598
Nov  9 05:37:34.598: INFO: >>> kubeConfig: /home/prow/go/src/k8s.io/windows-testing/capz-conf-tvyqls.kubeconfig
I1109 05:37:34.599425   14108 discovery.go:214] Invalidating discovery information
STEP: Building a namespace api object, basename horizontal-pod-autoscaling 11/09/22 05:37:34.599
I1109 05:37:34.638045   14108 reflector.go:221] Starting reflector *v1.ServiceAccount (0s) from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
I1109 05:37:34.638265   14108 reflector.go:257] Listing and watching *v1.ServiceAccount from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
I1109 05:37:34.700708   14108 reflector.go:227] Stopping reflector *v1.ServiceAccount (0s) from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
STEP: Waiting for a default service account to be provisioned in namespace 11/09/22 05:37:34.701
I1109 05:37:34.701321   14108 reflector.go:221] Starting reflector *v1.ServiceAccount (0s) from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
I1109 05:37:34.701335   14108 reflector.go:257] Listing and watching *v1.ServiceAccount from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
I1109 05:37:34.764426   14108 reflector.go:227] Stopping reflector *v1.ServiceAccount (0s) from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
STEP: Waiting for kube-root-ca.crt to be provisioned in namespace 11/09/22 05:37:34.764
I1109 05:37:34.764719   14108 reflector.go:221] Starting reflector *v1.ConfigMap (0s) from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
I1109 05:37:34.764742   14108 reflector.go:257] Listing and watching *v1.ConfigMap from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
I1109 05:37:34.826518   14108 reflector.go:227] Stopping reflector *v1.ConfigMap (0s) from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
[BeforeEach] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: Memory)
  test/e2e/framework/metrics/init/init.go:31
[It] Should scale from 1 pod to 3 pods and then from 3 pods to 5 pods using Average Utilization for aggregation
  test/e2e/autoscaling/horizontal_pod_autoscaling.go:154
Nov  9 05:37:34.826: INFO: >>> kubeConfig: /home/prow/go/src/k8s.io/windows-testing/capz-conf-tvyqls.kubeconfig
STEP: Running consuming RC test-deployment via apps/v1beta2, Kind=Deployment with 1 replicas 11/09/22 05:37:34.827
STEP: Creating deployment test-deployment in namespace horizontal-pod-autoscaling-6070 11/09/22 05:37:34.873
I1109 05:37:34.927293   14108 runners.go:193] Created deployment with name: test-deployment, namespace: horizontal-pod-autoscaling-6070, replica count: 1
I1109 05:37:34.927401   14108 reflector.go:221] Starting reflector *v1.Pod (0s) from test/utils/pod_store.go:57
I1109 05:37:34.927419   14108 reflector.go:257] Listing and watching *v1.Pod from test/utils/pod_store.go:57
I1109 05:37:44.981556   14108 runners.go:193] test-deployment Pods: 1 out of 1 created, 1 running, 0 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
STEP: Running controller 11/09/22 05:37:44.981
I1109 05:37:44.981921   14108 reflector.go:227] Stopping reflector *v1.Pod (0s) from test/utils/pod_store.go:57
STEP: creating replication controller test-deployment-ctrl in namespace horizontal-pod-autoscaling-6070 11/09/22 05:37:45.025
I1109 05:37:45.062430   14108 runners.go:193] Created replication controller with name: test-deployment-ctrl, namespace: horizontal-pod-autoscaling-6070, replica count: 1
I1109 05:37:45.062534   14108 reflector.go:221] Starting reflector *v1.Pod (0s) from test/utils/pod_store.go:57
I1109 05:37:45.062544   14108 reflector.go:257] Listing and watching *v1.Pod from test/utils/pod_store.go:57
I1109 05:37:55.113079   14108 runners.go:193] test-deployment-ctrl Pods: 1 out of 1 created, 1 running, 0 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1109 05:37:55.113435   14108 reflector.go:227] Stopping reflector *v1.Pod (0s) from test/utils/pod_store.go:57
Nov  9 05:38:00.113: INFO: Waiting for amount of service:test-deployment-ctrl endpoints to be 1
Nov  9 05:38:00.145: INFO: RC test-deployment: consume 0 millicores in total
Nov  9 05:38:00.146: INFO: RC test-deployment: consume 250 MB in total
Nov  9 05:38:00.146: INFO: RC test-deployment: setting consumption to 250 MB in total
Nov  9 05:38:00.146: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 05:38:00.146: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6070/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 05:38:00.146: INFO: RC test-deployment: consume custom metric 0 in total
Nov  9 05:38:00.146: INFO: RC test-deployment: disabling CPU consumption
Nov  9 05:38:00.146: INFO: RC test-deployment: disabling consumption of custom metric QPS
Nov  9 05:38:00.213: INFO: waiting for 3 replicas (current: 1)
Nov  9 05:38:20.246: INFO: waiting for 3 replicas (current: 2)
Nov  9 05:38:30.216: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 05:38:30.216: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6070/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 05:38:40.249: INFO: waiting for 3 replicas (current: 2)
Nov  9 05:39:00.247: INFO: waiting for 3 replicas (current: 2)
Nov  9 05:39:00.262: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 05:39:00.262: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6070/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
I1109 05:39:17.319836   14108 reflector.go:559] test/e2e/node/taints.go:147: Watch close - *v1.Pod total 6 items received
Nov  9 05:39:20.246: INFO: waiting for 3 replicas (current: 2)
Nov  9 05:39:30.301: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 05:39:30.301: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6070/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 05:39:40.250: INFO: waiting for 3 replicas (current: 2)
I1109 05:39:55.958285   14108 reflector.go:559] test/e2e/node/taints.go:147: Watch close - *v1.Pod total 11 items received
Nov  9 05:40:00.246: INFO: waiting for 3 replicas (current: 2)
Nov  9 05:40:00.351: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 05:40:00.351: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6070/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 05:40:20.248: INFO: waiting for 3 replicas (current: 2)
Nov  9 05:40:30.396: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 05:40:30.396: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6070/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 05:40:40.246: INFO: waiting for 3 replicas (current: 2)
Nov  9 05:41:00.246: INFO: waiting for 3 replicas (current: 2)
Nov  9 05:41:00.444: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 05:41:00.445: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6070/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 05:41:20.247: INFO: waiting for 3 replicas (current: 2)
Nov  9 05:41:30.490: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 05:41:30.490: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6070/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 05:41:40.247: INFO: waiting for 3 replicas (current: 2)
Nov  9 05:42:00.246: INFO: waiting for 3 replicas (current: 2)
Nov  9 05:42:00.537: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 05:42:00.537: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6070/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 05:42:20.245: INFO: waiting for 3 replicas (current: 2)
Nov  9 05:42:30.582: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 05:42:30.582: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6070/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 05:42:40.250: INFO: waiting for 3 replicas (current: 2)
Nov  9 05:43:00.245: INFO: waiting for 3 replicas (current: 2)
Nov  9 05:43:00.629: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 05:43:00.629: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6070/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 05:43:20.246: INFO: waiting for 3 replicas (current: 2)
Nov  9 05:43:30.669: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 05:43:30.669: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6070/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 05:43:40.247: INFO: waiting for 3 replicas (current: 2)
Nov  9 05:44:00.247: INFO: waiting for 3 replicas (current: 2)
Nov  9 05:44:00.715: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 05:44:00.715: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6070/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 05:44:20.247: INFO: waiting for 3 replicas (current: 2)
Nov  9 05:44:30.759: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 05:44:30.760: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6070/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 05:44:40.249: INFO: waiting for 3 replicas (current: 2)
Nov  9 05:45:00.246: INFO: waiting for 3 replicas (current: 2)
Nov  9 05:45:00.804: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 05:45:00.805: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6070/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 05:45:20.246: INFO: waiting for 3 replicas (current: 2)
Nov  9 05:45:30.850: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 05:45:30.850: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6070/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 05:45:40.251: INFO: waiting for 3 replicas (current: 2)
I1109 05:45:58.351110   14108 reflector.go:559] test/e2e/node/taints.go:147: Watch close - *v1.Pod total 8 items received
Nov  9 05:46:00.246: INFO: waiting for 3 replicas (current: 2)
Nov  9 05:46:00.896: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 05:46:00.896: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6070/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 05:46:20.247: INFO: waiting for 3 replicas (current: 2)
Nov  9 05:46:30.941: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 05:46:30.941: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6070/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 05:46:40.249: INFO: waiting for 3 replicas (current: 2)
Nov  9 05:47:00.247: INFO: waiting for 3 replicas (current: 2)
Nov  9 05:47:00.987: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 05:47:00.987: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6070/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 05:47:20.252: INFO: waiting for 3 replicas (current: 2)
Nov  9 05:47:31.032: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 05:47:31.032: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6070/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 05:47:40.250: INFO: waiting for 3 replicas (current: 2)
Nov  9 05:48:00.246: INFO: waiting for 3 replicas (current: 2)
Nov  9 05:48:01.079: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 05:48:01.079: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6070/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
I1109 05:48:05.989408   14108 reflector.go:559] test/e2e/node/taints.go:147: Watch close - *v1.Pod total 10 items received
Nov  9 05:48:20.247: INFO: waiting for 3 replicas (current: 2)
Nov  9 05:48:31.120: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 05:48:31.120: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6070/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 05:48:40.250: INFO: waiting for 3 replicas (current: 2)
Nov  9 05:49:00.246: INFO: waiting for 3 replicas (current: 2)
Nov  9 05:49:01.165: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 05:49:01.166: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6070/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 05:49:20.247: INFO: waiting for 3 replicas (current: 2)
Nov  9 05:49:31.210: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 05:49:31.211: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6070/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 05:49:40.247: INFO: waiting for 3 replicas (current: 2)
Nov  9 05:50:00.246: INFO: waiting for 3 replicas (current: 2)
Nov  9 05:50:01.259: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 05:50:01.260: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6070/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 05:50:20.251: INFO: waiting for 3 replicas (current: 2)
Nov  9 05:50:31.304: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 05:50:31.304: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6070/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 05:50:40.246: INFO: waiting for 3 replicas (current: 2)
Nov  9 05:51:00.246: INFO: waiting for 3 replicas (current: 2)
Nov  9 05:51:01.351: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 05:51:01.351: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6070/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 05:51:20.247: INFO: waiting for 3 replicas (current: 2)
Nov  9 05:51:31.396: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 05:51:31.396: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6070/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 05:51:40.250: INFO: waiting for 3 replicas (current: 2)
Nov  9 05:52:00.246: INFO: waiting for 3 replicas (current: 2)
Nov  9 05:52:01.444: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 05:52:01.444: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6070/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 05:52:20.250: INFO: waiting for 3 replicas (current: 2)
Nov  9 05:52:31.490: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 05:52:31.490: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6070/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 05:52:40.250: INFO: waiting for 3 replicas (current: 2)
Nov  9 05:53:00.245: INFO: waiting for 3 replicas (current: 2)
Nov  9 05:53:00.278: INFO: waiting for 3 replicas (current: 2)
Nov  9 05:53:00.278: INFO: Unexpected error: timeout waiting 15m0s for 3 replicas: 
    <*errors.errorString | 0xc000287c80>: {
        s: "timed out waiting for the condition",
    }
Nov  9 05:53:00.278: FAIL: timeout waiting 15m0s for 3 replicas: timed out waiting for the condition

Full Stack Trace
k8s.io/kubernetes/test/e2e/autoscaling.(*HPAScaleTest).run(0xc0031dde68, {0x74e3c28?, 0xc001fa4180?}, {{0x74b92e8, 0x4}, {0x74c2479, 0x7}, {0x74ca886, 0xa}}, 0xc00082ce10)
	test/e2e/autoscaling/horizontal_pod_autoscaling.go:209 +0x2d8
k8s.io/kubernetes/test/e2e/autoscaling.scaleUp({0x74e3c28?, 0x61dc905?}, {{0x74b92e8, 0x4}, {0x74c2479, 0x7}, {0x74ca886, 0xa}}, {0x74bdff0, 0x6}, ...)
	test/e2e/autoscaling/horizontal_pod_autoscaling.go:249 +0x212
k8s.io/kubernetes/test/e2e/autoscaling.glob..func7.1.1()
	test/e2e/autoscaling/horizontal_pod_autoscaling.go:155 +0x88
STEP: Removing consuming RC test-deployment 11/09/22 05:53:00.314
Nov  9 05:53:00.314: INFO: RC test-deployment: stopping metric consumer
Nov  9 05:53:00.315: INFO: RC test-deployment: stopping CPU consumer
Nov  9 05:53:00.315: INFO: RC test-deployment: stopping mem consumer
STEP: deleting Deployment.apps test-deployment in namespace horizontal-pod-autoscaling-6070, will wait for the garbage collector to delete the pods 11/09/22 05:53:10.315
I1109 05:53:10.350798   14108 reflector.go:221] Starting reflector *v1.Pod (0s) from test/utils/pod_store.go:57
I1109 05:53:10.350841   14108 reflector.go:257] Listing and watching *v1.Pod from test/utils/pod_store.go:57
Nov  9 05:53:10.487: INFO: Deleting Deployment.apps test-deployment took: 36.069854ms
I1109 05:53:10.588154   14108 resource.go:655] Ignoring inactive pod horizontal-pod-autoscaling-6070/test-deployment-669bb6996d-khhq7 in state Running, deletion time 2022-11-09 05:53:40 +0000 UTC
I1109 05:53:10.588207   14108 resource.go:655] Ignoring inactive pod horizontal-pod-autoscaling-6070/test-deployment-669bb6996d-kjc77 in state Running, deletion time 2022-11-09 05:53:40 +0000 UTC
Nov  9 05:53:10.588: INFO: Terminating Deployment.apps test-deployment pods took: 100.978076ms
I1109 05:53:12.789005   14108 reflector.go:227] Stopping reflector *v1.Pod (0s) from test/utils/pod_store.go:57
STEP: deleting ReplicationController test-deployment-ctrl in namespace horizontal-pod-autoscaling-6070, will wait for the garbage collector to delete the pods 11/09/22 05:53:12.86
I1109 05:53:12.894360   14108 reflector.go:221] Starting reflector *v1.Pod (0s) from test/utils/pod_store.go:57
I1109 05:53:12.894390   14108 reflector.go:257] Listing and watching *v1.Pod from test/utils/pod_store.go:57
Nov  9 05:53:12.983: INFO: Deleting ReplicationController test-deployment-ctrl took: 38.372795ms
I1109 05:53:13.083899   14108 resource.go:655] Ignoring inactive pod horizontal-pod-autoscaling-6070/test-deployment-ctrl-dk52h in state Running, deletion time 2022-11-09 05:53:13 +0000 UTC
Nov  9 05:53:13.083: INFO: Terminating ReplicationController test-deployment-ctrl pods took: 100.669787ms
I1109 05:53:14.985356   14108 reflector.go:227] Stopping reflector *v1.Pod (0s) from test/utils/pod_store.go:57
[AfterEach] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: Memory)
  test/e2e/framework/node/init/init.go:32
Nov  9 05:53:15.060: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
[DeferCleanup (Each)] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: Memory)
  test/e2e/framework/metrics/init/init.go:33
[DeferCleanup (Each)] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: Memory)
  dump namespaces | framework.go:196
STEP: dump namespace information after failure 11/09/22 05:53:15.098
STEP: Collecting events from namespace "horizontal-pod-autoscaling-6070". 11/09/22 05:53:15.098
STEP: Found 22 events. 11/09/22 05:53:15.135
Nov  9 05:53:15.135: INFO: At 2022-11-09 05:37:34 +0000 UTC - event for test-deployment: {deployment-controller } ScalingReplicaSet: Scaled up replica set test-deployment-669bb6996d to 1
Nov  9 05:53:15.135: INFO: At 2022-11-09 05:37:34 +0000 UTC - event for test-deployment-669bb6996d: {replicaset-controller } SuccessfulCreate: Created pod: test-deployment-669bb6996d-kjc77
Nov  9 05:53:15.135: INFO: At 2022-11-09 05:37:34 +0000 UTC - event for test-deployment-669bb6996d-kjc77: {default-scheduler } Scheduled: Successfully assigned horizontal-pod-autoscaling-6070/test-deployment-669bb6996d-kjc77 to capz-conf-hhzhh
Nov  9 05:53:15.135: INFO: At 2022-11-09 05:37:37 +0000 UTC - event for test-deployment-669bb6996d-kjc77: {kubelet capz-conf-hhzhh} Pulled: Container image "registry.k8s.io/e2e-test-images/resource-consumer:1.13" already present on machine
Nov  9 05:53:15.135: INFO: At 2022-11-09 05:37:37 +0000 UTC - event for test-deployment-669bb6996d-kjc77: {kubelet capz-conf-hhzhh} Created: Created container test-deployment
Nov  9 05:53:15.135: INFO: At 2022-11-09 05:37:39 +0000 UTC - event for test-deployment-669bb6996d-kjc77: {kubelet capz-conf-hhzhh} Started: Started container test-deployment
Nov  9 05:53:15.135: INFO: At 2022-11-09 05:37:45 +0000 UTC - event for test-deployment-ctrl: {replication-controller } SuccessfulCreate: Created pod: test-deployment-ctrl-dk52h
Nov  9 05:53:15.135: INFO: At 2022-11-09 05:37:45 +0000 UTC - event for test-deployment-ctrl-dk52h: {default-scheduler } Scheduled: Successfully assigned horizontal-pod-autoscaling-6070/test-deployment-ctrl-dk52h to capz-conf-m97cx
Nov  9 05:53:15.135: INFO: At 2022-11-09 05:37:47 +0000 UTC - event for test-deployment-ctrl-dk52h: {kubelet capz-conf-m97cx} Created: Created container test-deployment-ctrl
Nov  9 05:53:15.135: INFO: At 2022-11-09 05:37:47 +0000 UTC - event for test-deployment-ctrl-dk52h: {kubelet capz-conf-m97cx} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.40" already present on machine
Nov  9 05:53:15.135: INFO: At 2022-11-09 05:37:48 +0000 UTC - event for test-deployment-ctrl-dk52h: {kubelet capz-conf-m97cx} Started: Started container test-deployment-ctrl
Nov  9 05:53:15.135: INFO: At 2022-11-09 05:38:15 +0000 UTC - event for test-deployment: {horizontal-pod-autoscaler } SuccessfulRescale: New size: 2; reason: memory resource utilization (percentage of request) above target
Nov  9 05:53:15.135: INFO: At 2022-11-09 05:38:15 +0000 UTC - event for test-deployment: {deployment-controller } ScalingReplicaSet: Scaled up replica set test-deployment-669bb6996d to 2 from 1
Nov  9 05:53:15.135: INFO: At 2022-11-09 05:38:15 +0000 UTC - event for test-deployment-669bb6996d: {replicaset-controller } SuccessfulCreate: Created pod: test-deployment-669bb6996d-khhq7
Nov  9 05:53:15.135: INFO: At 2022-11-09 05:38:15 +0000 UTC - event for test-deployment-669bb6996d-khhq7: {default-scheduler } Scheduled: Successfully assigned horizontal-pod-autoscaling-6070/test-deployment-669bb6996d-khhq7 to capz-conf-m97cx
Nov  9 05:53:15.135: INFO: At 2022-11-09 05:38:17 +0000 UTC - event for test-deployment-669bb6996d-khhq7: {kubelet capz-conf-m97cx} Created: Created container test-deployment
Nov  9 05:53:15.135: INFO: At 2022-11-09 05:38:17 +0000 UTC - event for test-deployment-669bb6996d-khhq7: {kubelet capz-conf-m97cx} Pulled: Container image "registry.k8s.io/e2e-test-images/resource-consumer:1.13" already present on machine
Nov  9 05:53:15.135: INFO: At 2022-11-09 05:38:18 +0000 UTC - event for test-deployment-669bb6996d-khhq7: {kubelet capz-conf-m97cx} Started: Started container test-deployment
Nov  9 05:53:15.135: INFO: At 2022-11-09 05:53:10 +0000 UTC - event for test-deployment: {endpoint-controller } FailedToUpdateEndpoint: Failed to update endpoint horizontal-pod-autoscaling-6070/test-deployment: Operation cannot be fulfilled on endpoints "test-deployment": the object has been modified; please apply your changes to the latest version and try again
Nov  9 05:53:15.135: INFO: At 2022-11-09 05:53:10 +0000 UTC - event for test-deployment-669bb6996d-khhq7: {kubelet capz-conf-m97cx} Killing: Stopping container test-deployment
Nov  9 05:53:15.135: INFO: At 2022-11-09 05:53:10 +0000 UTC - event for test-deployment-669bb6996d-kjc77: {kubelet capz-conf-hhzhh} Killing: Stopping container test-deployment
Nov  9 05:53:15.135: INFO: At 2022-11-09 05:53:13 +0000 UTC - event for test-deployment-ctrl-dk52h: {kubelet capz-conf-m97cx} Killing: Stopping container test-deployment-ctrl
Nov  9 05:53:15.168: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
Nov  9 05:53:15.168: INFO: 
Nov  9 05:53:15.203: INFO: 
Logging node info for node capz-conf-hhzhh
Nov  9 05:53:15.236: INFO: Node Info: &Node{ObjectMeta:{capz-conf-hhzhh    641b6838-eeea-48b0-9dc0-0ff9ab096f10 41684 0 2022-11-09 01:07:16 +0000 UTC <nil> <nil> map[agentpool:windowsgmsa beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:windows failure-domain.beta.kubernetes.io/region:eastus failure-domain.beta.kubernetes.io/zone:0 kubernetes.io/arch:amd64 kubernetes.io/hostname:capz-conf-hhzhh kubernetes.io/os:windows node.kubernetes.io/instance-type:Standard_D4s_v3 node.kubernetes.io/windows-build:10.0.20348 topology.kubernetes.io/region:eastus topology.kubernetes.io/zone:0] map[cluster.x-k8s.io/cluster-name:capz-conf-tvyqls cluster.x-k8s.io/cluster-namespace:default cluster.x-k8s.io/machine:capz-conf-tvyqls-md-win-7848cfdb84-9mln2 cluster.x-k8s.io/owner-kind:MachineSet cluster.x-k8s.io/owner-name:capz-conf-tvyqls-md-win-7848cfdb84 kubeadm.alpha.kubernetes.io/cri-socket:npipe:////./pipe/containerd-containerd node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:10.1.0.4/16 projectcalico.org/IPv4VXLANTunnelAddr:192.168.84.65 projectcalico.org/VXLANTunnelMACAddr:00:15:5d:5b:b6:ce volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubeadm Update v1 2022-11-09 01:07:16 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:kubeadm.alpha.kubernetes.io/cri-socket":{}}}} } {kubelet.exe Update v1 2022-11-09 01:07:16 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:node.kubernetes.io/windows-build":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {manager Update v1 2022-11-09 01:08:02 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:cluster.x-k8s.io/cluster-name":{},"f:cluster.x-k8s.io/cluster-namespace":{},"f:cluster.x-k8s.io/machine":{},"f:cluster.x-k8s.io/owner-kind":{},"f:cluster.x-k8s.io/owner-name":{}}}} } {kube-controller-manager Update v1 2022-11-09 01:08:06 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}}} } {Go-http-client Update v1 2022-11-09 01:08:32 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4VXLANTunnelAddr":{},"f:projectcalico.org/VXLANTunnelMACAddr":{}}}} status} {cluster-api-e2e Update v1 2022-11-09 01:10:08 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:agentpool":{}}}} } {e2e.test Update v1 2022-11-09 03:54:19 +0000 UTC FieldsV1 {"f:status":{"f:capacity":{"f:example.com/fakePTSRes":{}}}} status} {kubelet.exe Update v1 2022-11-09 05:48:34 +0000 UTC FieldsV1 {"f:status":{"f:allocatable":{"f:example.com/fakePTSRes":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:,DoNotUseExternalID:,ProviderID:azure:///subscriptions/===REDACTED===/resourceGroups/capz-conf-tvyqls/providers/Microsoft.Compute/virtualMachines/capz-conf-hhzhh,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-azure-disk: {{8 0} {<nil>} 8 DecimalSI},cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{136912564224 0} {<nil>} 133703676Ki BinarySI},example.com/fakePTSRes: {{10 0} {<nil>} 10 DecimalSI},memory: {{17179398144 0} {<nil>} 16776756Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-azure-disk: {{8 0} {<nil>} 8 DecimalSI},cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{123221307598 0} {<nil>} 123221307598 DecimalSI},example.com/fakePTSRes: {{10 0} {<nil>} 10 DecimalSI},memory: {{17074540544 0} {<nil>} 16674356Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-09 05:48:34 +0000 UTC,LastTransitionTime:2022-11-09 01:07:16 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-09 05:48:34 +0000 UTC,LastTransitionTime:2022-11-09 01:07:16 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-09 05:48:34 +0000 UTC,LastTransitionTime:2022-11-09 01:07:16 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-09 05:48:34 +0000 UTC,LastTransitionTime:2022-11-09 01:08:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status,},},Addresses:[]NodeAddress{NodeAddress{Type:Hostname,Address:capz-conf-hhzhh,},NodeAddress{Type:InternalIP,Address:10.1.0.4,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:capz-conf-hhzhh,SystemUUID:5F9EA3E7-22F1-406E-9AAC-69BA5DD83BF4,BootID:8,KernelVersion:10.0.20348.1129,OSImage:Windows Server 2022 Datacenter,ContainerRuntimeVersion:containerd://1.6.8,KubeletVersion:v1.26.0-alpha.3.425+2f2240400391ad,KubeProxyVersion:v1.26.0-alpha.3.425+2f2240400391ad,OperatingSystem:windows,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:307641217,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:221349084,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:218942479,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:183620591,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:181784786,},ContainerImage{Names:[ghcr.io/kubernetes-sigs/sig-windows/eventflow-logger@sha256:63bf2aa9db909d0d90fb5205abf7fb2a6d9a494b89cbd2508a42457dfc875505 ghcr.io/kubernetes-sigs/sig-windows/eventflow-logger:v0.1.0],SizeBytes:148054279,},ContainerImage{Names:[ghcr.io/kubernetes-sigs/sig-windows/csi-proxy@sha256:96b4144986319a747ba599892454be2737aae6005d96b8e13ed481321ac3afba ghcr.io/kubernetes-sigs/sig-windows/csi-proxy:v1.0.2],SizeBytes:123925937,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/resource-consumer@sha256:ba5e047a337e5d0709bc57df45b95b2c7f6f2794b290e4e24f7fc8980d60b25a registry.k8s.io/e2e-test-images/resource-consumer:1.13],SizeBytes:121182992,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6],SizeBytes:119397646,},ContainerImage{Names:[docker.io/sigwindowstools/kube-proxy:v1.23.1-calico-hostprocess docker.io/sigwindowstools/kube-proxy:v1.26.0-alpha.3.425_2f2240400391ad-calico-hostprocess],SizeBytes:116182072,},ContainerImage{Names:[docker.io/sigwindowstools/calico-install@sha256:97bc10aa5000a0ee1c842ac32771fe7a45a3a5ca507711bdf57ae2eb5f293e2b docker.io/sigwindowstools/calico-install:v3.23.0-hostprocess],SizeBytes:47258343,},ContainerImage{Names:[docker.io/sigwindowstools/calico-node@sha256:ea8b55bde9aed6a649582a6e21029577430661c743d94b3a5e93d57e648874a2 docker.io/sigwindowstools/calico-node:v3.23.0-hostprocess],SizeBytes:27005624,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Nov  9 05:53:15.237: INFO: 
Logging kubelet events for node capz-conf-hhzhh
Nov  9 05:53:15.271: INFO: 
Logging pods the kubelet thinks is on node capz-conf-hhzhh
Nov  9 05:53:15.327: INFO: containerd-logger-z56l7 started at 2022-11-09 01:07:16 +0000 UTC (0+1 container statuses recorded)
Nov  9 05:53:15.327: INFO: 	Container containerd-logger ready: true, restart count 0
Nov  9 05:53:15.327: INFO: kube-proxy-windows-ldnh5 started at 2022-11-09 01:07:16 +0000 UTC (0+1 container statuses recorded)
Nov  9 05:53:15.327: INFO: 	Container kube-proxy ready: true, restart count 0
Nov  9 05:53:15.327: INFO: calico-node-windows-jpm48 started at 2022-11-09 01:07:16 +0000 UTC (1+2 container statuses recorded)
Nov  9 05:53:15.327: INFO: 	Init container install-cni ready: true, restart count 0
Nov  9 05:53:15.327: INFO: 	Container calico-node-felix ready: true, restart count 1
Nov  9 05:53:15.327: INFO: 	Container calico-node-startup ready: true, restart count 0
Nov  9 05:53:15.327: INFO: csi-proxy-dkn8s started at 2022-11-09 01:08:06 +0000 UTC (0+1 container statuses recorded)
Nov  9 05:53:15.327: INFO: 	Container csi-proxy ready: true, restart count 0
Nov  9 05:53:15.522: INFO: 
Latency metrics for node capz-conf-hhzhh
Nov  9 05:53:15.522: INFO: 
Logging node info for node capz-conf-m97cx
Nov  9 05:53:15.556: INFO: Node Info: &Node{ObjectMeta:{capz-conf-m97cx    36965107-96dc-426b-924d-c291c94a2d04 41691 0 2022-11-09 01:07:00 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:windows failure-domain.beta.kubernetes.io/region:eastus failure-domain.beta.kubernetes.io/zone:0 kubernetes.io/arch:amd64 kubernetes.io/hostname:capz-conf-m97cx kubernetes.io/os:windows node.kubernetes.io/instance-type:Standard_D4s_v3 node.kubernetes.io/windows-build:10.0.20348 topology.kubernetes.io/region:eastus topology.kubernetes.io/zone:0] map[cluster.x-k8s.io/cluster-name:capz-conf-tvyqls cluster.x-k8s.io/cluster-namespace:default cluster.x-k8s.io/machine:capz-conf-tvyqls-md-win-7848cfdb84-cw8bk cluster.x-k8s.io/owner-kind:MachineSet cluster.x-k8s.io/owner-name:capz-conf-tvyqls-md-win-7848cfdb84 kubeadm.alpha.kubernetes.io/cri-socket:npipe:////./pipe/containerd-containerd node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:10.1.0.5/16 projectcalico.org/IPv4VXLANTunnelAddr:192.168.100.1 projectcalico.org/VXLANTunnelMACAddr:00:15:5d:ac:92:68 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubeadm Update v1 2022-11-09 01:07:00 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:kubeadm.alpha.kubernetes.io/cri-socket":{}}}} } {kubelet.exe Update v1 2022-11-09 01:07:00 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:node.kubernetes.io/windows-build":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-11-09 01:07:31 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}}} } {manager Update v1 2022-11-09 01:08:02 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:cluster.x-k8s.io/cluster-name":{},"f:cluster.x-k8s.io/cluster-namespace":{},"f:cluster.x-k8s.io/machine":{},"f:cluster.x-k8s.io/owner-kind":{},"f:cluster.x-k8s.io/owner-name":{}}}} } {Go-http-client Update v1 2022-11-09 01:08:08 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4VXLANTunnelAddr":{},"f:projectcalico.org/VXLANTunnelMACAddr":{}}}} status} {e2e.test Update v1 2022-11-09 03:54:19 +0000 UTC FieldsV1 {"f:status":{"f:capacity":{"f:example.com/fakePTSRes":{},"f:example.com/fakecpu":{}}}} status} {kubelet.exe Update v1 2022-11-09 05:48:38 +0000 UTC FieldsV1 {"f:status":{"f:allocatable":{"f:example.com/fakePTSRes":{},"f:example.com/fakecpu":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:,DoNotUseExternalID:,ProviderID:azure:///subscriptions/===REDACTED===/resourceGroups/capz-conf-tvyqls/providers/Microsoft.Compute/virtualMachines/capz-conf-m97cx,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-azure-disk: {{8 0} {<nil>} 8 DecimalSI},cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{136912564224 0} {<nil>} 133703676Ki BinarySI},example.com/fakePTSRes: {{10 0} {<nil>} 10 DecimalSI},example.com/fakecpu: {{1 3} {<nil>} 1k DecimalSI},memory: {{17179398144 0} {<nil>} 16776756Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-azure-disk: {{8 0} {<nil>} 8 DecimalSI},cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{123221307598 0} {<nil>} 123221307598 DecimalSI},example.com/fakePTSRes: {{10 0} {<nil>} 10 DecimalSI},example.com/fakecpu: {{1 3} {<nil>} 1k DecimalSI},memory: {{17074540544 0} {<nil>} 16674356Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-09 05:48:38 +0000 UTC,LastTransitionTime:2022-11-09 01:07:00 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-09 05:48:38 +0000 UTC,LastTransitionTime:2022-11-09 01:07:00 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-09 05:48:38 +0000 UTC,LastTransitionTime:2022-11-09 01:07:00 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-09 05:48:38 +0000 UTC,LastTransitionTime:2022-11-09 01:07:31 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status,},},Addresses:[]NodeAddress{NodeAddress{Type:Hostname,Address:capz-conf-m97cx,},NodeAddress{Type:InternalIP,Address:10.1.0.5,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:capz-conf-m97cx,SystemUUID:FC504DDA-1329-4BBA-8C1E-77FAD783BCA1,BootID:8,KernelVersion:10.0.20348.1129,OSImage:Windows Server 2022 Datacenter,ContainerRuntimeVersion:containerd://1.6.8,KubeletVersion:v1.26.0-alpha.3.425+2f2240400391ad,KubeProxyVersion:v1.26.0-alpha.3.425+2f2240400391ad,OperatingSystem:windows,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:307641217,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:221349084,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:218942479,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:183620591,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:181784786,},ContainerImage{Names:[ghcr.io/kubernetes-sigs/sig-windows/eventflow-logger@sha256:63bf2aa9db909d0d90fb5205abf7fb2a6d9a494b89cbd2508a42457dfc875505 ghcr.io/kubernetes-sigs/sig-windows/eventflow-logger:v0.1.0],SizeBytes:148054279,},ContainerImage{Names:[ghcr.io/kubernetes-sigs/sig-windows/csi-proxy@sha256:96b4144986319a747ba599892454be2737aae6005d96b8e13ed481321ac3afba ghcr.io/kubernetes-sigs/sig-windows/csi-proxy:v1.0.2],SizeBytes:123925937,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/resource-consumer@sha256:ba5e047a337e5d0709bc57df45b95b2c7f6f2794b290e4e24f7fc8980d60b25a registry.k8s.io/e2e-test-images/resource-consumer:1.13],SizeBytes:121182992,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6],SizeBytes:119397646,},ContainerImage{Names:[docker.io/sigwindowstools/kube-proxy:v1.23.1-calico-hostprocess docker.io/sigwindowstools/kube-proxy:v1.26.0-alpha.3.425_2f2240400391ad-calico-hostprocess],SizeBytes:116182072,},ContainerImage{Names:[docker.io/sigwindowstools/calico-install@sha256:97bc10aa5000a0ee1c842ac32771fe7a45a3a5ca507711bdf57ae2eb5f293e2b docker.io/sigwindowstools/calico-install:v3.23.0-hostprocess],SizeBytes:47258343,},ContainerImage{Names:[docker.io/sigwindowstools/calico-node@sha256:ea8b55bde9aed6a649582a6e21029577430661c743d94b3a5e93d57e648874a2 docker.io/sigwindowstools/calico-node:v3.23.0-hostprocess],SizeBytes:27005624,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Nov  9 05:53:15.556: INFO: 
Logging kubelet events for node capz-conf-m97cx
Nov  9 05:53:15.600: INFO: 
Logging pods the kubelet thinks is on node capz-conf-m97cx
Nov  9 05:53:15.655: INFO: calico-node-windows-w5q9w started at 2022-11-09 01:07:01 +0000 UTC (1+2 container statuses recorded)
Nov  9 05:53:15.655: INFO: 	Init container install-cni ready: true, restart count 0
Nov  9 05:53:15.655: INFO: 	Container calico-node-felix ready: true, restart count 1
Nov  9 05:53:15.655: INFO: 	Container calico-node-startup ready: true, restart count 0
Nov  9 05:53:15.655: INFO: containerd-logger-v9qh5 started at 2022-11-09 01:07:01 +0000 UTC (0+1 container statuses recorded)
Nov  9 05:53:15.655: INFO: 	Container containerd-logger ready: true, restart count 0
Nov  9 05:53:15.655: INFO: csi-proxy-m7d9z started at 2022-11-09 05:13:44 +0000 UTC (0+1 container statuses recorded)
Nov  9 05:53:15.655: INFO: 	Container csi-proxy ready: true, restart count 0
Nov  9 05:53:15.655: INFO: kube-proxy-windows-55jdx started at 2022-11-09 01:07:01 +0000 UTC (0+1 container statuses recorded)
Nov  9 05:53:15.655: INFO: 	Container kube-proxy ready: true, restart count 0
Nov  9 05:53:15.823: INFO: 
Latency metrics for node capz-conf-m97cx
Nov  9 05:53:15.823: INFO: 
Logging node info for node capz-conf-tvyqls-control-plane-7vbm8
Nov  9 05:53:15.858: INFO: Node Info: &Node{ObjectMeta:{capz-conf-tvyqls-control-plane-7vbm8    59067037-dec8-4e79-8a8a-d150ea7ce6ba 41649 0 2022-11-09 01:04:38 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D2s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eastus failure-domain.beta.kubernetes.io/zone:eastus-3 kubernetes.io/arch:amd64 kubernetes.io/hostname:capz-conf-tvyqls-control-plane-7vbm8 kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:Standard_D2s_v3 topology.kubernetes.io/region:eastus topology.kubernetes.io/zone:eastus-3] map[cluster.x-k8s.io/cluster-name:capz-conf-tvyqls cluster.x-k8s.io/cluster-namespace:default cluster.x-k8s.io/machine:capz-conf-tvyqls-control-plane-ljqrh cluster.x-k8s.io/owner-kind:KubeadmControlPlane cluster.x-k8s.io/owner-name:capz-conf-tvyqls-control-plane kubeadm.alpha.kubernetes.io/cri-socket:unix:///var/run/containerd/containerd.sock node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:10.0.0.4/16 projectcalico.org/IPv4VXLANTunnelAddr:192.168.2.192 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-11-09 01:04:38 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kubeadm Update v1 2022-11-09 01:04:40 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:kubeadm.alpha.kubernetes.io/cri-socket":{}},"f:labels":{"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {manager Update v1 2022-11-09 01:04:55 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:cluster.x-k8s.io/cluster-name":{},"f:cluster.x-k8s.io/cluster-namespace":{},"f:cluster.x-k8s.io/machine":{},"f:cluster.x-k8s.io/owner-kind":{},"f:cluster.x-k8s.io/owner-name":{}}}} } {kube-controller-manager Update v1 2022-11-09 01:05:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:taints":{}}} } {Go-http-client Update v1 2022-11-09 01:05:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4VXLANTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-09 05:48:13 +0000 UTC FieldsV1 {"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:,DoNotUseExternalID:,ProviderID:azure:///subscriptions/===REDACTED===/resourceGroups/capz-conf-tvyqls/providers/Microsoft.Compute/virtualMachines/capz-conf-tvyqls-control-plane-7vbm8,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-azure-disk: {{4 0} {<nil>} 4 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{133003395072 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8344723456 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-azure-disk: {{4 0} {<nil>} 4 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{119703055367 0} {<nil>} 119703055367 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8239865856 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-09 01:05:27 +0000 UTC,LastTransitionTime:2022-11-09 01:05:27 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-09 05:48:13 +0000 UTC,LastTransitionTime:2022-11-09 01:04:17 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-09 05:48:13 +0000 UTC,LastTransitionTime:2022-11-09 01:04:17 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-09 05:48:13 +0000 UTC,LastTransitionTime:2022-11-09 01:04:17 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-09 05:48:13 +0000 UTC,LastTransitionTime:2022-11-09 01:05:22 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:Hostname,Address:capz-conf-tvyqls-control-plane-7vbm8,},NodeAddress{Type:InternalIP,Address:10.0.0.4,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:37a2e1ceaa0d455a8de0dda7f144eb7f,SystemUUID:50325a86-a564-f642-b7cf-2b4ce49e7328,BootID:49adcb05-103e-4672-b80c-d3eea5cf97ca,KernelVersion:5.4.0-1091-azure,OSImage:Ubuntu 18.04.6 LTS,ContainerRuntimeVersion:containerd://1.6.2,KubeletVersion:v1.26.0-alpha.3.425+2f2240400391ad,KubeProxyVersion:v1.26.0-alpha.3.425+2f2240400391ad,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[docker.io/calico/cni@sha256:914823d144204288f881e49b93b6852febfe669074cd4e2a782860981615f521 docker.io/calico/cni:v3.23.0],SizeBytes:110494683,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:b83c1d70989e1fe87583607bf5aee1ee34e52773d4755b95f5cf5a451962f3a4 registry.k8s.io/etcd:3.5.5-0],SizeBytes:102417044,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:102157811,},ContainerImage{Names:[docker.io/calico/node@sha256:4763820ecb4d8e82483a2ffabfec7fcded9603318692df210a778d223a4d7474 docker.io/calico/node:v3.23.0],SizeBytes:71573794,},ContainerImage{Names:[docker.io/calico/kube-controllers@sha256:78bc199299f966b0694dc4044501aee2d7ebd6862b2b0a00bca3ee8d3813c82f docker.io/calico/kube-controllers:v3.23.0],SizeBytes:56343954,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:51155161,},ContainerImage{Names:[gcr.io/k8s-staging-ci-images/kube-apiserver@sha256:f4a034110c7fecfbd53047bb4e7a568b1d123694426129781f718b2b83f650dd gcr.io/k8s-staging-ci-images/kube-apiserver:v1.26.0-alpha.3.425_2f2240400391ad],SizeBytes:34983033,},ContainerImage{Names:[registry.k8s.io/kube-apiserver@sha256:4188262a351f156e8027ff81693d771c35b34b668cbd61e59c4a4490dd5c08f3 registry.k8s.io/kube-apiserver:v1.25.3],SizeBytes:34238163,},ContainerImage{Names:[gcr.io/k8s-staging-ci-images/kube-controller-manager@sha256:4660397406f51cc97599ef65f46644d7cffad440bc3169fdfc0224626b3b2780 gcr.io/k8s-staging-ci-images/kube-controller-manager:v1.26.0-alpha.3.425_2f2240400391ad],SizeBytes:31999085,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager@sha256:d3a06262256f3e7578d5f77df137a8cdf58f9f498f35b5b56d116e8a7e31dc91 registry.k8s.io/kube-controller-manager:v1.25.3],SizeBytes:31261869,},ContainerImage{Names:[k8s.gcr.io/metrics-server/metrics-server@sha256:6385aec64bb97040a5e692947107b81e178555c7a5b71caa90d733e4130efc10 k8s.gcr.io/metrics-server/metrics-server:v0.5.2],SizeBytes:26023008,},ContainerImage{Names:[gcr.io/k8s-staging-ci-images/kube-proxy@sha256:8dcde74eedeada05ac0d0b3fec95ca4299544d64c362e7968f7f5ecb69186030 gcr.io/k8s-staging-ci-images/kube-proxy:v1.26.0-alpha.3.425_2f2240400391ad],SizeBytes:21362412,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:6bf25f038543e1f433cb7f2bdda445ed348c7b9279935ebc2ae4f432308ed82f registry.k8s.io/kube-proxy:v1.25.3],SizeBytes:20265805,},ContainerImage{Names:[gcr.io/k8s-staging-ci-images/kube-scheduler@sha256:fb955dc0dd068e8c525ac60042dac1c9cd349272855639394d761125851f8397 gcr.io/k8s-staging-ci-images/kube-scheduler:v1.26.0-alpha.3.425_2f2240400391ad],SizeBytes:17278982,},ContainerImage{Names:[registry.k8s.io/kube-scheduler@sha256:f478aa916568b00269068ff1e9ff742ecc16192eb6e371e30f69f75df904162e registry.k8s.io/kube-scheduler:v1.25.3],SizeBytes:15798744,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[docker.io/sigwindowstools/k8s-gmsa-webhook@sha256:56bba3b5a4788c05dc45f51444b924f6bed05a29789e04e43a6e2c3dbb07d9ce docker.io/sigwindowstools/k8s-gmsa-webhook:latest],SizeBytes:8874257,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Nov  9 05:53:15.859: INFO: 
Logging kubelet events for node capz-conf-tvyqls-control-plane-7vbm8
Nov  9 05:53:15.892: INFO: 
Logging pods the kubelet thinks is on node capz-conf-tvyqls-control-plane-7vbm8
Nov  9 05:53:15.953: INFO: calico-node-7bphx started at 2022-11-09 01:05:02 +0000 UTC (2+1 container statuses recorded)
Nov  9 05:53:15.953: INFO: 	Init container upgrade-ipam ready: true, restart count 0
Nov  9 05:53:15.953: INFO: 	Init container install-cni ready: true, restart count 0
Nov  9 05:53:15.953: INFO: 	Container calico-node ready: true, restart count 0
Nov  9 05:53:15.953: INFO: coredns-fdff55fb9-vl42q started at 2022-11-09 01:10:12 +0000 UTC (0+1 container statuses recorded)
Nov  9 05:53:15.953: INFO: 	Container coredns ready: true, restart count 0
Nov  9 05:53:15.953: INFO: etcd-capz-conf-tvyqls-control-plane-7vbm8 started at 2022-11-09 01:04:42 +0000 UTC (0+1 container statuses recorded)
Nov  9 05:53:15.953: INFO: 	Container etcd ready: true, restart count 0
Nov  9 05:53:15.953: INFO: kube-apiserver-capz-conf-tvyqls-control-plane-7vbm8 started at 2022-11-09 01:04:42 +0000 UTC (0+1 container statuses recorded)
Nov  9 05:53:15.953: INFO: 	Container kube-apiserver ready: true, restart count 0
Nov  9 05:53:15.953: INFO: kube-controller-manager-capz-conf-tvyqls-control-plane-7vbm8 started at 2022-11-09 01:04:42 +0000 UTC (0+1 container statuses recorded)
Nov  9 05:53:15.953: INFO: 	Container kube-controller-manager ready: true, restart count 0
Nov  9 05:53:15.953: INFO: kube-scheduler-capz-conf-tvyqls-control-plane-7vbm8 started at 2022-11-09 01:04:42 +0000 UTC (0+1 container statuses recorded)
Nov  9 05:53:15.953: INFO: 	Container kube-scheduler ready: true, restart count 0
Nov  9 05:53:15.953: INFO: kube-proxy-9ns2c started at 2022-11-09 01:04:42 +0000 UTC (0+1 container statuses recorded)
Nov  9 05:53:15.953: INFO: 	Container kube-proxy ready: true, restart count 0
Nov  9 05:53:15.953: INFO: calico-kube-controllers-56c5ff4bf8-gbkvc started at 2022-11-09 01:05:22 +0000 UTC (0+1 container statuses recorded)
Nov  9 05:53:15.953: INFO: 	Container calico-kube-controllers ready: true, restart count 0
Nov  9 05:53:15.953: INFO: metrics-server-954b56d74-7hmq4 started at 2022-11-09 01:05:30 +0000 UTC (0+1 container statuses recorded)
Nov  9 05:53:15.953: INFO: 	Container metrics-server ready: true, restart count 0
Nov  9 05:53:15.953: INFO: coredns-fdff55fb9-4wwnr started at 2022-11-09 01:10:12 +0000 UTC (0+1 container statuses recorded)
Nov  9 05:53:15.953: INFO: 	Container coredns ready: true, restart count 0
Nov  9 05:53:16.151: INFO: 
Latency metrics for node capz-conf-tvyqls-control-plane-7vbm8
[DeferCleanup (Each)] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: Memory)
  tear down framework | framework.go:193
STEP: Destroying namespace "horizontal-pod-autoscaling-6070" for this suite. 11/09/22 05:53:16.151

				Click to see stdout/stderrfrom junit_01.xml

Filter through log files | View test history on testgrid

Kubernetes e2e suite [It] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: Memory) [Serial] [Slow] Deployment (Pod Resource) Should scale from 1 pod to 3 pods and then from 3 pods to 5 pods using Average Value for aggregation 15m40s

go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\s\[sig\-autoscaling\]\s\[Feature\:HPA\]\sHorizontal\spod\sautoscaling\s\(scale\sresource\:\sMemory\)\s\[Serial\]\s\[Slow\]\sDeployment\s\(Pod\sResource\)\sShould\sscale\sfrom\s1\spod\sto\s3\spods\sand\sthen\sfrom\s3\spods\sto\s5\spods\susing\sAverage\sValue\sfor\saggregation$'

test/e2e/autoscaling/horizontal_pod_autoscaling.go:209
k8s.io/kubernetes/test/e2e/autoscaling.(*HPAScaleTest).run(0xc005b6be68, {0x74e3c28?, 0xc00387eea0?}, {{0x74b92e8, 0x4}, {0x74c2479, 0x7}, {0x74ca886, 0xa}}, 0xc00082ce10)
	test/e2e/autoscaling/horizontal_pod_autoscaling.go:209 +0x2d8
k8s.io/kubernetes/test/e2e/autoscaling.scaleUp({0x74e3c28?, 0x61dc905?}, {{0x74b92e8, 0x4}, {0x74c2479, 0x7}, {0x74ca886, 0xa}}, {0x74bdff0, 0x6}, ...)
	test/e2e/autoscaling/horizontal_pod_autoscaling.go:249 +0x212
k8s.io/kubernetes/test/e2e/autoscaling.glob..func7.1.2()
	test/e2e/autoscaling/horizontal_pod_autoscaling.go:158 +0x88
				[BeforeEach] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: Memory)
  set up framework | framework.go:178
STEP: Creating a kubernetes client 11/09/22 03:04:44.289
Nov  9 03:04:44.289: INFO: >>> kubeConfig: /home/prow/go/src/k8s.io/windows-testing/capz-conf-tvyqls.kubeconfig
I1109 03:04:44.290604   14108 discovery.go:214] Invalidating discovery information
STEP: Building a namespace api object, basename horizontal-pod-autoscaling 11/09/22 03:04:44.29
I1109 03:04:44.329565   14108 reflector.go:221] Starting reflector *v1.ServiceAccount (0s) from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
I1109 03:04:44.329601   14108 reflector.go:257] Listing and watching *v1.ServiceAccount from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
I1109 03:04:44.391523   14108 reflector.go:227] Stopping reflector *v1.ServiceAccount (0s) from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
STEP: Waiting for a default service account to be provisioned in namespace 11/09/22 03:04:44.391
I1109 03:04:44.391794   14108 reflector.go:221] Starting reflector *v1.ServiceAccount (0s) from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
I1109 03:04:44.391806   14108 reflector.go:257] Listing and watching *v1.ServiceAccount from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
I1109 03:04:44.454471   14108 reflector.go:227] Stopping reflector *v1.ServiceAccount (0s) from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
STEP: Waiting for kube-root-ca.crt to be provisioned in namespace 11/09/22 03:04:44.454
I1109 03:04:44.454697   14108 reflector.go:221] Starting reflector *v1.ConfigMap (0s) from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
I1109 03:04:44.454709   14108 reflector.go:257] Listing and watching *v1.ConfigMap from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
I1109 03:04:44.516206   14108 reflector.go:227] Stopping reflector *v1.ConfigMap (0s) from vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146
[BeforeEach] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: Memory)
  test/e2e/framework/metrics/init/init.go:31
[It] Should scale from 1 pod to 3 pods and then from 3 pods to 5 pods using Average Value for aggregation
  test/e2e/autoscaling/horizontal_pod_autoscaling.go:157
Nov  9 03:04:44.516: INFO: >>> kubeConfig: /home/prow/go/src/k8s.io/windows-testing/capz-conf-tvyqls.kubeconfig
STEP: Running consuming RC test-deployment via apps/v1beta2, Kind=Deployment with 1 replicas 11/09/22 03:04:44.517
STEP: Creating deployment test-deployment in namespace horizontal-pod-autoscaling-6862 11/09/22 03:04:44.561
I1109 03:04:44.598313   14108 runners.go:193] Created deployment with name: test-deployment, namespace: horizontal-pod-autoscaling-6862, replica count: 1
I1109 03:04:44.598399   14108 reflector.go:221] Starting reflector *v1.Pod (0s) from test/utils/pod_store.go:57
I1109 03:04:44.598409   14108 reflector.go:257] Listing and watching *v1.Pod from test/utils/pod_store.go:57
I1109 03:04:54.650623   14108 runners.go:193] test-deployment Pods: 1 out of 1 created, 1 running, 0 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
STEP: Running controller 11/09/22 03:04:54.65
I1109 03:04:54.651061   14108 reflector.go:227] Stopping reflector *v1.Pod (0s) from test/utils/pod_store.go:57
STEP: creating replication controller test-deployment-ctrl in namespace horizontal-pod-autoscaling-6862 11/09/22 03:04:54.695
I1109 03:04:54.733318   14108 runners.go:193] Created replication controller with name: test-deployment-ctrl, namespace: horizontal-pod-autoscaling-6862, replica count: 1
I1109 03:04:54.733436   14108 reflector.go:221] Starting reflector *v1.Pod (0s) from test/utils/pod_store.go:57
I1109 03:04:54.733448   14108 reflector.go:257] Listing and watching *v1.Pod from test/utils/pod_store.go:57
I1109 03:05:04.785561   14108 runners.go:193] test-deployment-ctrl Pods: 1 out of 1 created, 1 running, 0 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1109 03:05:04.785848   14108 reflector.go:227] Stopping reflector *v1.Pod (0s) from test/utils/pod_store.go:57
Nov  9 03:05:09.786: INFO: Waiting for amount of service:test-deployment-ctrl endpoints to be 1
Nov  9 03:05:09.818: INFO: RC test-deployment: consume 0 millicores in total
Nov  9 03:05:09.818: INFO: RC test-deployment: disabling CPU consumption
Nov  9 03:05:09.818: INFO: RC test-deployment: consume 250 MB in total
Nov  9 03:05:09.818: INFO: RC test-deployment: setting consumption to 250 MB in total
Nov  9 03:05:09.818: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 03:05:09.818: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6862/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 03:05:09.818: INFO: RC test-deployment: consume custom metric 0 in total
Nov  9 03:05:09.818: INFO: RC test-deployment: disabling consumption of custom metric QPS
Nov  9 03:05:09.885: INFO: waiting for 3 replicas (current: 1)
Nov  9 03:05:29.917: INFO: waiting for 3 replicas (current: 1)
Nov  9 03:05:39.889: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 03:05:39.890: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6862/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 03:05:49.918: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:06:09.918: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:06:10.937: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 03:06:10.937: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6862/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 03:06:29.919: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:06:40.977: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 03:06:40.977: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6862/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 03:06:49.919: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:07:09.918: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:07:11.020: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 03:07:11.020: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6862/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 03:07:29.921: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:07:41.083: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 03:07:41.083: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6862/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 03:07:49.921: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:08:09.920: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:08:11.128: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 03:08:11.128: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6862/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 03:08:29.923: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:08:41.173: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 03:08:41.173: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6862/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 03:08:49.921: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:09:09.918: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:09:11.218: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 03:09:11.218: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6862/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 03:09:29.918: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:09:41.269: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 03:09:41.269: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6862/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 03:09:49.918: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:10:09.920: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:10:11.315: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 03:10:11.315: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6862/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 03:10:29.922: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:10:41.356: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 03:10:41.357: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6862/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 03:10:49.922: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:11:09.919: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:11:11.403: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 03:11:11.403: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6862/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 03:11:29.922: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:11:41.449: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 03:11:41.449: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6862/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 03:11:49.922: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:12:09.919: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:12:11.495: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 03:12:11.495: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6862/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 03:12:29.918: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:12:41.542: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 03:12:41.542: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6862/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 03:12:49.922: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:13:09.918: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:13:11.587: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 03:13:11.587: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6862/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 03:13:29.922: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:13:41.631: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 03:13:41.631: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6862/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 03:13:49.920: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:14:09.918: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:14:11.678: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 03:14:11.678: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6862/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 03:14:29.922: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:14:41.721: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 03:14:41.721: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6862/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 03:14:49.922: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:15:09.920: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:15:11.766: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 03:15:11.766: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6862/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 03:15:29.922: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:15:41.807: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 03:15:41.807: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6862/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 03:15:49.921: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:16:09.918: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:16:11.854: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 03:16:11.854: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6862/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 03:16:29.922: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:16:41.900: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 03:16:41.900: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6862/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 03:16:49.919: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:17:09.918: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:17:11.946: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 03:17:11.946: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6862/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 03:17:29.918: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:17:41.992: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 03:17:41.992: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6862/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 03:17:49.918: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:18:09.918: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:18:12.036: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 03:18:12.036: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6862/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 03:18:29.921: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:18:42.083: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 03:18:42.083: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6862/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 03:18:49.921: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:19:09.918: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:19:12.128: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 03:19:12.128: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6862/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 03:19:29.918: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:19:42.175: INFO: RC test-deployment: sending request to consume 250 MB
Nov  9 03:19:42.175: INFO: ConsumeMem URL: {https   capz-conf-tvyqls-1ee29e45.eastus.cloudapp.azure.com:6443 /api/v1/namespaces/horizontal-pod-autoscaling-6862/services/test-deployment-ctrl/proxy/ConsumeMem  false false durationSec=30&megabytes=250&requestSizeMegabytes=100  }
Nov  9 03:19:49.922: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:20:09.919: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:20:09.951: INFO: waiting for 3 replicas (current: 2)
Nov  9 03:20:09.951: INFO: Unexpected error: timeout waiting 15m0s for 3 replicas: 
    <*errors.errorString | 0xc000287c80>: {
        s: "timed out waiting for the condition",
    }
Nov  9 03:20:09.951: FAIL: timeout waiting 15m0s for 3 replicas: timed out waiting for the condition

Full Stack Trace
k8s.io/kubernetes/test/e2e/autoscaling.(*HPAScaleTest).run(0xc005b6be68, {0x74e3c28?, 0xc00387eea0?}, {{0x74b92e8, 0x4}, {0x74c2479, 0x7}, {0x74ca886, 0xa}}, 0xc00082ce10)
	test/e2e/autoscaling/horizontal_pod_autoscaling.go:209 +0x2d8
k8s.io/kubernetes/test/e2e/autoscaling.scaleUp({0x74e3c28?, 0x61dc905?}, {{0x74b92e8, 0x4}, {0x74c2479, 0x7}, {0x74ca886, 0xa}}, {0x74bdff0, 0x6}, ...)
	test/e2e/autoscaling/horizontal_pod_autoscaling.go:249 +0x212
k8s.io/kubernetes/test/e2e/autoscaling.glob..func7.1.2()
	test/e2e/autoscaling/horizontal_pod_autoscaling.go:158 +0x88
STEP: Removing consuming RC test-deployment 11/09/22 03:20:09.991
Nov  9 03:20:09.991: INFO: RC test-deployment: stopping metric consumer
Nov  9 03:20:09.991: INFO: RC test-deployment: stopping CPU consumer
Nov  9 03:20:09.991: INFO: RC test-deployment: stopping mem consumer
STEP: deleting Deployment.apps test-deployment in namespace horizontal-pod-autoscaling-6862, will wait for the garbage collector to delete the pods 11/09/22 03:20:19.992
I1109 03:20:20.025328   14108 reflector.go:221] Starting reflector *v1.Pod (0s) from test/utils/pod_store.go:57
I1109 03:20:20.025398   14108 reflector.go:257] Listing and watching *v1.Pod from test/utils/pod_store.go:57
Nov  9 03:20:20.111: INFO: Deleting Deployment.apps test-deployment took: 35.660363ms
I1109 03:20:20.212010   14108 resource.go:655] Ignoring inactive pod horizontal-pod-autoscaling-6862/test-deployment-669bb6996d-kh4mc in state Running, deletion time 2022-11-09 03:20:50 +0000 UTC
I1109 03:20:20.212066   14108 resource.go:655] Ignoring inactive pod horizontal-pod-autoscaling-6862/test-deployment-669bb6996d-8xhzm in state Running, deletion time 2022-11-09 03:20:50 +0000 UTC
Nov  9 03:20:20.212: INFO: Terminating Deployment.apps test-deployment pods took: 100.335578ms
I1109 03:20:22.013282   14108 reflector.go:227] Stopping reflector *v1.Pod (0s) from test/utils/pod_store.go:57
STEP: deleting ReplicationController test-deployment-ctrl in namespace horizontal-pod-autoscaling-6862, will wait for the garbage collector to delete the pods 11/09/22 03:20:22.066
I1109 03:20:22.099297   14108 reflector.go:221] Starting reflector *v1.Pod (0s) from test/utils/pod_store.go:57
I1109 03:20:22.099337   14108 reflector.go:257] Listing and watching *v1.Pod from test/utils/pod_store.go:57
Nov  9 03:20:22.184: INFO: Deleting ReplicationController test-deployment-ctrl took: 35.010513ms
I1109 03:20:22.285542   14108 resource.go:655] Ignoring inactive pod horizontal-pod-autoscaling-6862/test-deployment-ctrl-7w4q6 in state Running, deletion time 2022-11-09 03:20:23 +0000 UTC
Nov  9 03:20:22.285: INFO: Terminating ReplicationController test-deployment-ctrl pods took: 101.472188ms
I1109 03:20:23.687347   14108 reflector.go:227] Stopping reflector *v1.Pod (0s) from test/utils/pod_store.go:57
[AfterEach] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: Memory)
  test/e2e/framework/node/init/init.go:32
Nov  9 03:20:23.747: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
[DeferCleanup (Each)] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: Memory)
  test/e2e/framework/metrics/init/init.go:33
[DeferCleanup (Each)] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: Memory)
  dump namespaces | framework.go:196
STEP: dump namespace information after failure 11/09/22 03:20:23.784
STEP: Collecting events from namespace "horizontal-pod-autoscaling-6862". 11/09/22 03:20:23.784
STEP: Found 21 events. 11/09/22 03:20:23.819
Nov  9 03:20:23.819: INFO: At 2022-11-09 03:04:44 +0000 UTC - event for test-deployment: {deployment-controller } ScalingReplicaSet: Scaled up replica set test-deployment-669bb6996d to 1
Nov  9 03:20:23.819: INFO: At 2022-11-09 03:04:44 +0000 UTC - event for test-deployment-669bb6996d: {replicaset-controller } SuccessfulCreate: Created pod: test-deployment-669bb6996d-8xhzm
Nov  9 03:20:23.819: INFO: At 2022-11-09 03:04:44 +0000 UTC - event for test-deployment-669bb6996d-8xhzm: {default-scheduler } Scheduled: Successfully assigned horizontal-pod-autoscaling-6862/test-deployment-669bb6996d-8xhzm to capz-conf-hhzhh
Nov  9 03:20:23.819: INFO: At 2022-11-09 03:04:48 +0000 UTC - event for test-deployment-669bb6996d-8xhzm: {kubelet capz-conf-hhzhh} Pulled: Container image "registry.k8s.io/e2e-test-images/resource-consumer:1.13" already present on machine
Nov  9 03:20:23.819: INFO: At 2022-11-09 03:04:49 +0000 UTC - event for test-deployment-669bb6996d-8xhzm: {kubelet capz-conf-hhzhh} Created: Created container test-deployment
Nov  9 03:20:23.819: INFO: At 2022-11-09 03:04:52 +0000 UTC - event for test-deployment-669bb6996d-8xhzm: {kubelet capz-conf-hhzhh} Started: Started container test-deployment
Nov  9 03:20:23.819: INFO: At 2022-11-09 03:04:54 +0000 UTC - event for test-deployment-ctrl: {replication-controller } SuccessfulCreate: Created pod: test-deployment-ctrl-7w4q6
Nov  9 03:20:23.819: INFO: At 2022-11-09 03:04:54 +0000 UTC - event for test-deployment-ctrl-7w4q6: {default-scheduler } Scheduled: Successfully assigned horizontal-pod-autoscaling-6862/test-deployment-ctrl-7w4q6 to capz-conf-m97cx
Nov  9 03:20:23.819: INFO: At 2022-11-09 03:04:58 +0000 UTC - event for test-deployment-ctrl-7w4q6: {kubelet capz-conf-m97cx} Created: Created container test-deployment-ctrl
Nov  9 03:20:23.819: INFO: At 2022-11-09 03:04:58 +0000 UTC - event for test-deployment-ctrl-7w4q6: {kubelet capz-conf-m97cx} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.40" already present on machine
Nov  9 03:20:23.819: INFO: At 2022-11-09 03:05:00 +0000 UTC - event for test-deployment-ctrl-7w4q6: {kubelet capz-conf-m97cx} Started: Started container test-deployment-ctrl
Nov  9 03:20:23.819: INFO: At 2022-11-09 03:05:24 +0000 UTC - event for test-deployment: {horizontal-pod-autoscaler } SuccessfulRescale: New size: 2; reason: memory resource above target
Nov  9 03:20:23.819: INFO: At 2022-11-09 03:05:24 +0000 UTC - event for test-deployment: {deployment-controller } ScalingReplicaSet: Scaled up replica set test-deployment-669bb6996d to 2 from 1
Nov  9 03:20:23.819: INFO: At 2022-11-09 03:05:24 +0000 UTC - event for test-deployment-669bb6996d: {replicaset-controller } SuccessfulCreate: Created pod: test-deployment-669bb6996d-kh4mc
Nov  9 03:20:23.819: INFO: At 2022-11-09 03:05:24 +0000 UTC - event for test-deployment-669bb6996d-kh4mc: {default-scheduler } Scheduled: Successfully assigned horizontal-pod-autoscaling-6862/test-deployment-669bb6996d-kh4mc to capz-conf-m97cx
Nov  9 03:20:23.819: INFO: At 2022-11-09 03:05:28 +0000 UTC - event for test-deployment-669bb6996d-kh4mc: {kubelet capz-conf-m97cx} Created: Created container test-deployment
Nov  9 03:20:23.819: INFO: At 2022-11-09 03:05:28 +0000 UTC - event for test-deployment-669bb6996d-kh4mc: {kubelet capz-conf-m97cx} Pulled: Container image "registry.k8s.io/e2e-test-images/resource-consumer:1.13" already present on machine
Nov  9 03:20:23.819: INFO: At 2022-11-09 03:05:31 +0000 UTC - event for test-deployment-669bb6996d-kh4mc: {kubelet capz-conf-m97cx} Started: Started container test-deployment
Nov  9 03:20:23.819: INFO: At 2022-11-09 03:20:20 +0000 UTC - event for test-deployment-669bb6996d-8xhzm: {kubelet capz-conf-hhzhh} Killing: Stopping container test-deployment
Nov  9 03:20:23.819: INFO: At 2022-11-09 03:20:20 +0000 UTC - event for test-deployment-669bb6996d-kh4mc: {kubelet capz-conf-m97cx} Killing: Stopping container test-deployment
Nov  9 03:20:23.819: INFO: At 2022-11-09 03:20:22 +0000 UTC - event for test-deployment-ctrl-7w4q6: {kubelet capz-conf-m97cx} Killing: Stopping container test-deployment-ctrl
Nov  9 03:20:23.850: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
Nov  9 03:20:23.850: INFO: 
Nov  9 03:20:23.885: INFO: 
Logging node info for node capz-conf-hhzhh
Nov  9 03:20:23.918: INFO: Node Info: &Node{ObjectMeta:{capz-conf-hhzhh    641b6838-eeea-48b0-9dc0-0ff9ab096f10 18926 0 2022-11-09 01:07:16 +0000 UTC <nil> <nil> map[agentpool:windowsgmsa beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:windows failure-domain.beta.kubernetes.io/region:eastus failure-domain.beta.kubernetes.io/zone:0 kubernetes.io/arch:amd64 kubernetes.io/hostname:capz-conf-hhzhh kubernetes.io/os:windows node.kubernetes.io/instance-type:Standard_D4s_v3 node.kubernetes.io/windows-build:10.0.20348 topology.kubernetes.io/region:eastus topology.kubernetes.io/zone:0] map[cluster.x-k8s.io/cluster-name:capz-conf-tvyqls cluster.x-k8s.io/cluster-namespace:default cluster.x-k8s.io/machine:capz-conf-tvyqls-md-win-7848cfdb84-9mln2 cluster.x-k8s.io/owner-kind:MachineSet cluster.x-k8s.io/owner-name:capz-conf-tvyqls-md-win-7848cfdb84 kubeadm.alpha.kubernetes.io/cri-socket:npipe:////./pipe/containerd-containerd node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:10.1.0.4/16 projectcalico.org/IPv4VXLANTunnelAddr:192.168.84.65 projectcalico.org/VXLANTunnelMACAddr:00:15:5d:5b:b6:ce volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubeadm Update v1 2022-11-09 01:07:16 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:kubeadm.alpha.kubernetes.io/cri-socket":{}}}} } {kubelet.exe Update v1 2022-11-09 01:07:16 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:node.kubernetes.io/windows-build":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {manager Update v1 2022-11-09 01:08:02 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:cluster.x-k8s.io/cluster-name":{},"f:cluster.x-k8s.io/cluster-namespace":{},"f:cluster.x-k8s.io/machine":{},"f:cluster.x-k8s.io/owner-kind":{},"f:cluster.x-k8s.io/owner-name":{}}}} } {kube-controller-manager Update v1 2022-11-09 01:08:06 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}}} } {Go-http-client Update v1 2022-11-09 01:08:32 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4VXLANTunnelAddr":{},"f:projectcalico.org/VXLANTunnelMACAddr":{}}}} status} {cluster-api-e2e Update v1 2022-11-09 01:10:08 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:agentpool":{}}}} } {e2e.test Update v1 2022-11-09 01:22:08 +0000 UTC FieldsV1 {"f:status":{"f:capacity":{"f:example.com/fakePTSRes":{}}}} status} {kubelet.exe Update v1 2022-11-09 03:19:40 +0000 UTC FieldsV1 {"f:status":{"f:allocatable":{"f:example.com/fakePTSRes":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:,DoNotUseExternalID:,ProviderID:azure:///subscriptions/===REDACTED===/resourceGroups/capz-conf-tvyqls/providers/Microsoft.Compute/virtualMachines/capz-conf-hhzhh,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-azure-disk: {{8 0} {<nil>} 8 DecimalSI},cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{136912564224 0} {<nil>} 133703676Ki BinarySI},example.com/fakePTSRes: {{10 0} {<nil>} 10 DecimalSI},memory: {{17179398144 0} {<nil>} 16776756Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-azure-disk: {{8 0} {<nil>} 8 DecimalSI},cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{123221307598 0} {<nil>} 123221307598 DecimalSI},example.com/fakePTSRes: {{10 0} {<nil>} 10 DecimalSI},memory: {{17074540544 0} {<nil>} 16674356Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-09 03:19:40 +0000 UTC,LastTransitionTime:2022-11-09 01:07:16 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-09 03:19:40 +0000 UTC,LastTransitionTime:2022-11-09 01:07:16 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-09 03:19:40 +0000 UTC,LastTransitionTime:2022-11-09 01:07:16 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-09 03:19:40 +0000 UTC,LastTransitionTime:2022-11-09 01:08:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status,},},Addresses:[]NodeAddress{NodeAddress{Type:Hostname,Address:capz-conf-hhzhh,},NodeAddress{Type:InternalIP,Address:10.1.0.4,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:capz-conf-hhzhh,SystemUUID:5F9EA3E7-22F1-406E-9AAC-69BA5DD83BF4,BootID:8,KernelVersion:10.0.20348.1129,OSImage:Windows Server 2022 Datacenter,ContainerRuntimeVersion:containerd://1.6.8,KubeletVersion:v1.26.0-alpha.3.425+2f2240400391ad,KubeProxyVersion:v1.26.0-alpha.3.425+2f2240400391ad,OperatingSystem:windows,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:307641217,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:221349084,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:218942479,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:183620591,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:181784786,},ContainerImage{Names:[ghcr.io/kubernetes-sigs/sig-windows/eventflow-logger@sha256:63bf2aa9db909d0d90fb5205abf7fb2a6d9a494b89cbd2508a42457dfc875505 ghcr.io/kubernetes-sigs/sig-windows/eventflow-logger:v0.1.0],SizeBytes:148054279,},ContainerImage{Names:[ghcr.io/kubernetes-sigs/sig-windows/csi-proxy@sha256:96b4144986319a747ba599892454be2737aae6005d96b8e13ed481321ac3afba ghcr.io/kubernetes-sigs/sig-windows/csi-proxy:v1.0.2],SizeBytes:123925937,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/resource-consumer@sha256:ba5e047a337e5d0709bc57df45b95b2c7f6f2794b290e4e24f7fc8980d60b25a registry.k8s.io/e2e-test-images/resource-consumer:1.13],SizeBytes:121182992,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6],SizeBytes:119397646,},ContainerImage{Names:[docker.io/sigwindowstools/kube-proxy:v1.23.1-calico-hostprocess docker.io/sigwindowstools/kube-proxy:v1.26.0-alpha.3.425_2f2240400391ad-calico-hostprocess],SizeBytes:116182072,},ContainerImage{Names:[docker.io/sigwindowstools/calico-install@sha256:97bc10aa5000a0ee1c842ac32771fe7a45a3a5ca507711bdf57ae2eb5f293e2b docker.io/sigwindowstools/calico-install:v3.23.0-hostprocess],SizeBytes:47258343,},ContainerImage{Names:[docker.io/sigwindowstools/calico-node@sha256:ea8b55bde9aed6a649582a6e21029577430661c743d94b3a5e93d57e648874a2 docker.io/sigwindowstools/calico-node:v3.23.0-hostprocess],SizeBytes:27005624,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Nov  9 03:20:23.918: INFO: 
Logging kubelet events for node capz-conf-hhzhh
Nov  9 03:20:23.950: INFO: 
Logging pods the kubelet thinks is on node capz-conf-hhzhh
Nov  9 03:20:24.002: INFO: containerd-logger-z56l7 started at 2022-11-09 01:07:16 +0000 UTC (0+1 container statuses recorded)
Nov  9 03:20:24.002: INFO: 	Container containerd-logger ready: true, restart count 0
Nov  9 03:20:24.002: INFO: kube-proxy-windows-ldnh5 started at 2022-11-09 01:07:16 +0000 UTC (0+1 container statuses recorded)
Nov  9 03:20:24.002: INFO: 	Container kube-proxy ready: true, restart count 0
Nov  9 03:20:24.002: INFO: calico-node-windows-jpm48 started at 2022-11-09 01:07:16 +0000 UTC (1+2 container statuses recorded)
Nov  9 03:20:24.002: INFO: 	Init container install-cni ready: true, restart count 0
Nov  9 03:20:24.002: INFO: 	Container calico-node-felix ready: true, restart count 1
Nov  9 03:20:24.002: INFO: 	Container calico-node-startup ready: true, restart count 0
Nov  9 03:20:24.002: INFO: csi-proxy-dkn8s started at 2022-11-09 01:08:06 +0000 UTC (0+1 container statuses recorded)
Nov  9 03:20:24.002: INFO: 	Container csi-proxy ready: true, restart count 0
Nov  9 03:20:24.189: INFO: 
Latency metrics for node capz-conf-hhzhh
Nov  9 03:20:24.189: INFO: 
Logging node info for node capz-conf-m97cx
Nov  9 03:20:24.221: INFO: Node Info: &Node{ObjectMeta:{capz-conf-m97cx    36965107-96dc-426b-924d-c291c94a2d04 18868 0 2022-11-09 01:07:00 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D4s_v3 beta.kubernetes.io/os:windows failure-domain.beta.kubernetes.io/region:eastus failure-domain.beta.kubernetes.io/zone:0 kubernetes.io/arch:amd64 kubernetes.io/hostname:capz-conf-m97cx kubernetes.io/os:windows node.kubernetes.io/instance-type:Standard_D4s_v3 node.kubernetes.io/windows-build:10.0.20348 topology.kubernetes.io/region:eastus topology.kubernetes.io/zone:0] map[cluster.x-k8s.io/cluster-name:capz-conf-tvyqls cluster.x-k8s.io/cluster-namespace:default cluster.x-k8s.io/machine:capz-conf-tvyqls-md-win-7848cfdb84-cw8bk cluster.x-k8s.io/owner-kind:MachineSet cluster.x-k8s.io/owner-name:capz-conf-tvyqls-md-win-7848cfdb84 kubeadm.alpha.kubernetes.io/cri-socket:npipe:////./pipe/containerd-containerd node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:10.1.0.5/16 projectcalico.org/IPv4VXLANTunnelAddr:192.168.100.1 projectcalico.org/VXLANTunnelMACAddr:00:15:5d:ac:92:68 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubeadm Update v1 2022-11-09 01:07:00 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:kubeadm.alpha.kubernetes.io/cri-socket":{}}}} } {kubelet.exe Update v1 2022-11-09 01:07:00 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:node.kubernetes.io/windows-build":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-11-09 01:07:31 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}}} } {manager Update v1 2022-11-09 01:08:02 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:cluster.x-k8s.io/cluster-name":{},"f:cluster.x-k8s.io/cluster-namespace":{},"f:cluster.x-k8s.io/machine":{},"f:cluster.x-k8s.io/owner-kind":{},"f:cluster.x-k8s.io/owner-name":{}}}} } {Go-http-client Update v1 2022-11-09 01:08:08 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4VXLANTunnelAddr":{},"f:projectcalico.org/VXLANTunnelMACAddr":{}}}} status} {e2e.test Update v1 2022-11-09 03:03:35 +0000 UTC FieldsV1 {"f:status":{"f:capacity":{"f:example.com/fakePTSRes":{},"f:example.com/fakecpu":{}}}} status} {kubelet.exe Update v1 2022-11-09 03:19:03 +0000 UTC FieldsV1 {"f:status":{"f:allocatable":{"f:example.com/fakePTSRes":{},"f:example.com/fakecpu":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:,DoNotUseExternalID:,ProviderID:azure:///subscriptions/===REDACTED===/resourceGroups/capz-conf-tvyqls/providers/Microsoft.Compute/virtualMachines/capz-conf-m97cx,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-azure-disk: {{8 0} {<nil>} 8 DecimalSI},cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{136912564224 0} {<nil>} 133703676Ki BinarySI},example.com/fakePTSRes: {{10 0} {<nil>} 10 DecimalSI},example.com/fakecpu: {{1 3} {<nil>} 1k DecimalSI},memory: {{17179398144 0} {<nil>} 16776756Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-azure-disk: {{8 0} {<nil>} 8 DecimalSI},cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{123221307598 0} {<nil>} 123221307598 DecimalSI},example.com/fakePTSRes: {{10 0} {<nil>} 10 DecimalSI},example.com/fakecpu: {{1 3} {<nil>} 1k DecimalSI},memory: {{17074540544 0} {<nil>} 16674356Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-09 03:19:03 +0000 UTC,LastTransitionTime:2022-11-09 01:07:00 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-09 03:19:03 +0000 UTC,LastTransitionTime:2022-11-09 01:07:00 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-09 03:19:03 +0000 UTC,LastTransitionTime:2022-11-09 01:07:00 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-09 03:19:03 +0000 UTC,LastTransitionTime:2022-11-09 01:07:31 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status,},},Addresses:[]NodeAddress{NodeAddress{Type:Hostname,Address:capz-conf-m97cx,},NodeAddress{Type:InternalIP,Address:10.1.0.5,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:capz-conf-m97cx,SystemUUID:FC504DDA-1329-4BBA-8C1E-77FAD783BCA1,BootID:8,KernelVersion:10.0.20348.1129,OSImage:Windows Server 2022 Datacenter,ContainerRuntimeVersion:containerd://1.6.8,KubeletVersion:v1.26.0-alpha.3.425+2f2240400391ad,KubeProxyVersion:v1.26.0-alpha.3.425+2f2240400391ad,OperatingSystem:windows,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:307641217,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:221349084,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:218942479,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:183620591,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:181784786,},ContainerImage{Names:[ghcr.io/kubernetes-sigs/sig-windows/eventflow-logger@sha256:63bf2aa9db909d0d90fb5205abf7fb2a6d9a494b89cbd2508a42457dfc875505 ghcr.io/kubernetes-sigs/sig-windows/eventflow-logger:v0.1.0],SizeBytes:148054279,},ContainerImage{Names:[ghcr.io/kubernetes-sigs/sig-windows/csi-proxy@sha256:96b4144986319a747ba599892454be2737aae6005d96b8e13ed481321ac3afba ghcr.io/kubernetes-sigs/sig-windows/csi-proxy:v1.0.2],SizeBytes:123925937,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/resource-consumer@sha256:ba5e047a337e5d0709bc57df45b95b2c7f6f2794b290e4e24f7fc8980d60b25a registry.k8s.io/e2e-test-images/resource-consumer:1.13],SizeBytes:121182992,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6],SizeBytes:119397646,},ContainerImage{Names:[docker.io/sigwindowstools/kube-proxy:v1.23.1-calico-hostprocess docker.io/sigwindowstools/kube-proxy:v1.26.0-alpha.3.425_2f2240400391ad-calico-hostprocess],SizeBytes:116182072,},ContainerImage{Names:[docker.io/sigwindowstools/calico-install@sha256:97bc10aa5000a0ee1c842ac32771fe7a45a3a5ca507711bdf57ae2eb5f293e2b docker.io/sigwindowstools/calico-install:v3.23.0-hostprocess],SizeBytes:47258343,},ContainerImage{Names:[docker.io/sigwindowstools/calico-node@sha256:ea8b55bde9aed6a649582a6e21029577430661c743d94b3a5e93d57e648874a2 docker.io/sigwindowstools/calico-node:v3.23.0-hostprocess],SizeBytes:27005624,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Nov  9 03:20:24.222: INFO: 
Logging kubelet events for node capz-conf-m97cx
Nov  9 03:20:24.254: INFO: 
Logging pods the kubelet thinks is on node capz-conf-m97cx
Nov  9 03:20:24.306: INFO: kube-proxy-windows-55jdx started at 2022-11-09 01:07:01 +0000 UTC (0+1 container statuses recorded)
Nov  9 03:20:24.306: INFO: 	Container kube-proxy ready: true, restart count 0
Nov  9 03:20:24.306: INFO: csi-proxy-tn9x7 started at 2022-11-09 01:07:31 +0000 UTC (0+1 container statuses recorded)
Nov  9 03:20:24.306: INFO: 	Container csi-proxy ready: true, restart count 0
Nov  9 03:20:24.306: INFO: calico-node-windows-w5q9w started at 2022-11-09 01:07:01 +0000 UTC (1+2 container statuses recorded)
Nov  9 03:20:24.306: INFO: 	Init container install-cni ready: true, restart count 0
Nov  9 03:20:24.306: INFO: 	Container calico-node-felix ready: true, restart count 1
Nov  9 03:20:24.306: INFO: 	Container calico-node-startup ready: true, restart count 0
Nov  9 03:20:24.306: INFO: containerd-logger-v9qh5 started at 2022-11-09 01:07:01 +0000 UTC (0+1 container statuses recorded)
Nov  9 03:20:24.306: INFO: 	Container containerd-logger ready: true, restart count 0
Nov  9 03:20:24.467: INFO: 
Latency metrics for node capz-conf-m97cx
Nov  9 03:20:24.467: INFO: 
Logging node info for node capz-conf-tvyqls-control-plane-7vbm8
Nov  9 03:20:24.501: INFO: Node Info: &Node{ObjectMeta:{capz-conf-tvyqls-control-plane-7vbm8    59067037-dec8-4e79-8a8a-d150ea7ce6ba 18983 0 2022-11-09 01:04:38 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D2s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eastus failure-domain.beta.kubernetes.io/zone:eastus-3 kubernetes.io/arch:amd64 kubernetes.io/hostname:capz-conf-tvyqls-control-plane-7vbm8 kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:Standard_D2s_v3 topology.kubernetes.io/region:eastus topology.kubernetes.io/zone:eastus-3] map[cluster.x-k8s.io/cluster-name:capz-conf-tvyqls cluster.x-k8s.io/cluster-namespace:default cluster.x-k8s.io/machine:capz-conf-tvyqls-control-plane-ljqrh cluster.x-k8s.io/owner-kind:KubeadmControlPlane cluster.x-k8s.io/owner-name:capz-conf-tvyqls-control-plane kubeadm.alpha.kubernetes.io/cri-socket:unix:///var/run/containerd/containerd.sock node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:10.0.0.4/16 projectcalico.org/IPv4VXLANTunnelAddr:192.168.2.192 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-11-09 01:04:38 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kubeadm Update v1 2022-11-09 01:04:40 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:kubeadm.alpha.kubernetes.io/cri-socket":{}},"f:labels":{"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {manager Update v1 2022-11-09 01:04:55 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:cluster.x-k8s.io/cluster-name":{},"f:cluster.x-k8s.io/cluster-namespace":{},"f:cluster.x-k8s.io/machine":{},"f:cluster.x-k8s.io/owner-kind":{},"f:cluster.x-k8s.io/owner-name":{}}}} } {kube-controller-manager Update v1 2022-11-09 01:05:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:taints":{}}} } {Go-http-client Update v1 2022-11-09 01:05:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4VXLANTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-09 03:20:15 +0000 UTC FieldsV1 {"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:,DoNotUseExternalID:,ProviderID:azure:///subscriptions/===REDACTED===/resourceGroups/capz-conf-tvyqls/providers/Microsoft.Compute/virtualMachines/capz-conf-tvyqls-control-plane-7vbm8,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-azure-disk: {{4 0} {<nil>} 4 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{133003395072 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8344723456 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-azure-disk: {{4 0} {<nil>} 4 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{119703055367 0} {<nil>} 119703055367 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8239865856 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-09 01:05:27 +0000 UTC,LastTransitionTime:2022-11-09 01:05:27 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-09 03:20:15 +0000 UTC,LastTransitionTime:2022-11-09 01:04:17 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-09 03:20:15 +0000 UTC,LastTransitionTime:2022-11-09 01:04:17 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-09 03:20:15 +0000 UTC,LastTransitionTime:2022-11-09 01:04:17 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-09 03:20:15 +0000 UTC,LastTransitionTime:2022-11-09 01:05:22 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:Hostname,Address:capz-conf-tvyqls-control-plane-7vbm8,},NodeAddress{Type:InternalIP,Address:10.0.0.4,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:37a2e1ceaa0d455a8de0dda7f144eb7f,SystemUUID:50325a86-a564-f642-b7cf-2b4ce49e7328,BootID:49adcb05-103e-4672-b80c-d3eea5cf97ca,KernelVersion:5.4.0-1091-azure,OSImage:Ubuntu 18.04.6 LTS,ContainerRuntimeVersion:containerd://1.6.2,KubeletVersion:v1.26.0-alpha.3.425+2f2240400391ad,KubeProxyVersion:v1.26.0-alpha.3.425+2f2240400391ad,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[docker.io/calico/cni@sha256:914823d144204288f881e49b93b6852febfe669074cd4e2a782860981615f521 docker.io/calico/cni:v3.23.0],SizeBytes:110494683,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:b83c1d70989e1fe87583607bf5aee1ee34e52773d4755b95f5cf5a451962f3a4 registry.k8s.io/etcd:3.5.5-0],SizeBytes:102417044,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:102157811,},ContainerImage{Names:[docker.io/calico/node@sha256:4763820ecb4d8e82483a2ffabfec7fcded9603318692df210a778d223a4d7474 docker.io/calico/node:v3.23.0],SizeBytes:71573794,},ContainerImage{Names:[docker.io/calico/kube-controllers@sha256:78bc199299f966b0694dc4044501aee2d7ebd6862b2b0a00bca3ee8d3813c82f docker.io/calico/kube-controllers:v3.23.0],SizeBytes:56343954,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:51155161,},ContainerImage{Names:[gcr.io/k8s-staging-ci-images/kube-apiserver@sha256:f4a034110c7fecfbd53047bb4e7a568b1d123694426129781f718b2b83f650dd gcr.io/k8s-staging-ci-images/kube-apiserver:v1.26.0-alpha.3.425_2f2240400391ad],SizeBytes:34983033,},ContainerImage{Names:[registry.k8s.io/kube-apiserver@sha256:4188262a351f156e8027ff81693d771c35b34b668cbd61e59c4a4490dd5c08f3 registry.k8s.io/kube-apiserver:v1.25.3],SizeBytes:34238163,},ContainerImage{Names:[gcr.io/k8s-staging-ci-images/kube-controller-manager@sha256:4660397406f51cc97599ef65f46644d7cffad440bc3169fdfc0224626b3b2780 gcr.io/k8s-staging-ci-images/kube-controller-manager:v1.26.0-alpha.3.425_2f2240400391ad],SizeBytes:31999085,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager@sha256:d3a06262256f3e7578d5f77df137a8cdf58f9f498f35b5b56d116e8a7e31dc91 registry.k8s.io/kube-controller-manager:v1.25.3],SizeBytes:31261869,},ContainerImage{Names:[k8s.gcr.io/metrics-server/metrics-server@sha256:6385aec64bb97040a5e692947107b81e178555c7a5b71caa90d733e4130efc10 k8s.gcr.io/metrics-server/metrics-server:v0.5.2],SizeBytes:26023008,},ContainerImage{Names:[gcr.io/k8s-staging-ci-images/kube-proxy@sha256:8dcde74eedeada05ac0d0b3fec95ca4299544d64c362e7968f7f5ecb69186030 gcr.io/k8s-staging-ci-images/kube-proxy:v1.26.0-alpha.3.425_2f2240400391ad],SizeBytes:21362412,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:6bf25f038543e1f433cb7f2bdda445ed348c7b9279935ebc2ae4f432308ed82f registry.k8s.io/kube-proxy:v1.25.3],SizeBytes:20265805,},ContainerImage{Names:[gcr.io/k8s-staging-ci-images/kube-scheduler@sha256:fb955dc0dd068e8c525ac60042dac1c9cd349272855639394d761125851f8397 gcr.io/k8s-staging-ci-images/kube-scheduler:v1.26.0-alpha.3.425_2f2240400391ad],SizeBytes:17278982,},ContainerImage{Names:[registry.k8s.io/kube-scheduler@sha256:f478aa916568b00269068ff1e9ff742ecc16192eb6e371e30f69f75df904162e registry.k8s.io/kube-scheduler:v1.25.3],SizeBytes:15798744,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[docker.io/sigwindowstools/k8s-gmsa-webhook@sha256:56bba3b5a4788c05dc45f51444b924f6bed05a29789e04e43a6e2c3dbb07d9ce docker.io/sigwindowstools/k8s-gmsa-webhook:latest],SizeBytes:8874257,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Nov  9 03:20:24.501: INFO: 
Logging kubelet events for node capz-conf-tvyqls-control-plane-7vbm8
Nov  9 03:20:24.533: INFO: 
Logging pods the kubelet thinks is on node capz-conf-tvyqls-control-plane-7vbm8
Nov  9 03:20:24.587: INFO: kube-controller-manager-capz-conf-tvyqls-control-plane-7vbm8 started at 2022-11-09 01:04:42 +0000 UTC (0+1 container statuses recorded)
Nov  9 03:20:24.587: INFO: 	Container kube-controller-manager ready: true, restart count 0
Nov  9 03:20:24.587: INFO: kube-scheduler-capz-conf-tvyqls-control-plane-7vbm8 started at 2022-11-09 01:04:42 +0000 UTC (0+1 container statuses recorded)
Nov  9 03:20:24.587: INFO: 	Container kube-scheduler ready: true, restart count 0
Nov  9 03:20:24.587: INFO: kube-proxy-9ns2c started at 2022-11-09 01:04:42 +0000 UTC (0+1 container statuses recorded)
Nov  9 03:20:24.587: INFO: 	Container kube-proxy ready: true, restart count 0
Nov  9 03:20:24.587: INFO: calico-kube-controllers-56c5ff4bf8-gbkvc started at 2022-11-09 01:05:22 +0000 UTC (0+1 container statuses recorded)
Nov  9 03:20:24.587: INFO: 	Container calico-kube-controllers ready: true, restart count 0
Nov  9 03:20:24.587: INFO: metrics-server-954b56d74-7hmq4 started at 2022-11-09 01:05:30 +0000 UTC (0+1 container statuses recorded)
Nov  9 03:20:24.587: INFO: 	Container metrics-server ready: true, restart count 0
Nov  9 03:20:24.587: INFO: etcd-capz-conf-tvyqls-control-plane-7vbm8 started at 2022-11-09 01:04:42 +0000 UTC (0+1 container statuses recorded)
Nov  9 03:20:24.587: INFO: 	Container etcd ready: true, restart count 0
Nov  9 03:20:24.587: INFO: kube-apiserver-capz-conf-tvyqls-control-plane-7vbm8 started at 2022-11-09 01:04:42 +0000 UTC (0+1 container statuses recorded)
Nov  9 03:20:24.587: INFO: 	Container kube-apiserver ready: true, restart count 0
Nov  9 03:20:24.587: INFO: coredns-fdff55fb9-4wwnr started at 2022-11-09 01:10:12 +0000 UTC (0+1 container statuses recorded)
Nov  9 03:20:24.587: INFO: 	Container coredns ready: true, restart count 0
Nov  9 03:20:24.587: INFO: coredns-fdff55fb9-vl42q started at 2022-11-09 01:10:12 +0000 UTC (0+1 container statuses recorded)
Nov  9 03:20:24.587: INFO: 	Container coredns ready: true, restart count 0
Nov  9 03:20:24.587: INFO: calico-node-7bphx started at 2022-11-09 01:05:02 +0000 UTC (2+1 container statuses recorded)
Nov  9 03:20:24.587: INFO: 	Init container upgrade-ipam ready: true, restart count 0
Nov  9 03:20:24.587: INFO: 	Init container install-cni ready: true, restart count 0
Nov  9 03:20:24.587: INFO: 	Container calico-node ready: true, restart count 0
Nov  9 03:20:24.770: INFO: 
Latency metrics for node capz-conf-tvyqls-control-plane-7vbm8
[DeferCleanup (Each)] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: Memory)
  tear down framework | framework.go:193
STEP: Destroying namespace "horizontal-pod-autoscaling-6862" for this suite. 11/09/22 03:20:24.77

				Click to see stdout/stderrfrom junit_01.xml

Filter through log files | View test history on testgrid

Show 6703 Skipped Tests

ci-kubernetes-e2e-capz-master-containerd-windows-2022-serial-slow #1590144996969811968

Test Failures

Kubernetes e2e suite [It] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: CPU) [Serial] [Slow] Deployment (Pod Resource) Should scale from 1 pod to 3 pods and then from 3 pods to 5 pods using Average Utilization for aggregation 15m40s

Kubernetes e2e suite [It] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: CPU) [Serial] [Slow] Deployment (Pod Resource) Should scale from 1 pod to 3 pods and then from 3 pods to 5 pods using Average Value for aggregation 15m42s

Kubernetes e2e suite [It] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: CPU) [Serial] [Slow] ReplicaSet Should scale from 1 pod to 3 pods and then from 3 pods to 5 pods 15m40s

Kubernetes e2e suite [It] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: CPU) [Serial] [Slow] ReplicationController Should scale from 1 pod to 3 pods and then from 3 pods to 5 pods and verify decision stability 15m41s

Kubernetes e2e suite [It] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: Memory) [Serial] [Slow] Deployment (Pod Resource) Should scale from 1 pod to 3 pods and then from 3 pods to 5 pods using Average Utilization for aggregation 15m41s

Kubernetes e2e suite [It] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: Memory) [Serial] [Slow] Deployment (Pod Resource) Should scale from 1 pod to 3 pods and then from 3 pods to 5 pods using Average Value for aggregation 15m40s

Passed

Skipped