e2e-kops-grid-cilium-etcd-flatcar-k23-docker #1617188262932123648 Results

Test Failures

Kubernetes e2e suite [sig-cli] Kubectl client Guestbook application should create and stop a working application [Conformance] 3m21s

go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[sig\-cli\]\sKubectl\sclient\sGuestbook\sapplication\sshould\screate\sand\sstop\sa\sworking\sapplication\s\s\[Conformance\]$'

/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:633
Jan 22 16:06:30.772: Cannot added new entry in 180 seconds.
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl/kubectl.go:375
				[BeforeEach] [sig-cli] Kubectl client
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185
�[1mSTEP�[0m: Creating a kubernetes client
Jan 22 16:03:11.940: INFO: >>> kubeConfig: /root/.kube/config
�[1mSTEP�[0m: Building a namespace api object, basename kubectl
�[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace
�[1mSTEP�[0m: Waiting for kube-root-ca.crt to be provisioned in namespace
[BeforeEach] [sig-cli] Kubectl client
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl/kubectl.go:244
[It] should create and stop a working application  [Conformance]
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:633
�[1mSTEP�[0m: creating all guestbook components
Jan 22 16:03:12.170: INFO: apiVersion: v1
kind: Service
metadata:
  name: agnhost-replica
  labels:
    app: agnhost
    role: replica
    tier: backend
spec:
  ports:
  - port: 6379
  selector:
    app: agnhost
    role: replica
    tier: backend

Jan 22 16:03:12.170: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/aa65bb68-9a6c-11ed-a697-56ea552f9d82/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-etcd-flatcar-k23-docker.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-357 create -f -'
Jan 22 16:03:13.269: INFO: stderr: ""
Jan 22 16:03:13.269: INFO: stdout: "service/agnhost-replica created\n"
Jan 22 16:03:13.269: INFO: apiVersion: v1
kind: Service
metadata:
  name: agnhost-primary
  labels:
    app: agnhost
    role: primary
    tier: backend
spec:
  ports:
  - port: 6379
    targetPort: 6379
  selector:
    app: agnhost
    role: primary
    tier: backend

Jan 22 16:03:13.269: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/aa65bb68-9a6c-11ed-a697-56ea552f9d82/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-etcd-flatcar-k23-docker.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-357 create -f -'
Jan 22 16:03:13.605: INFO: stderr: ""
Jan 22 16:03:13.605: INFO: stdout: "service/agnhost-primary created\n"
Jan 22 16:03:13.605: INFO: apiVersion: v1
kind: Service
metadata:
  name: frontend
  labels:
    app: guestbook
    tier: frontend
spec:
  # if your cluster supports it, uncomment the following to automatically create
  # an external load-balanced IP for the frontend service.
  # type: LoadBalancer
  ports:
  - port: 80
  selector:
    app: guestbook
    tier: frontend

Jan 22 16:03:13.605: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/aa65bb68-9a6c-11ed-a697-56ea552f9d82/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-etcd-flatcar-k23-docker.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-357 create -f -'
Jan 22 16:03:13.950: INFO: stderr: ""
Jan 22 16:03:13.950: INFO: stdout: "service/frontend created\n"
Jan 22 16:03:13.950: INFO: apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend
spec:
  replicas: 3
  selector:
    matchLabels:
      app: guestbook
      tier: frontend
  template:
    metadata:
      labels:
        app: guestbook
        tier: frontend
    spec:
      containers:
      - name: guestbook-frontend
        image: k8s.gcr.io/e2e-test-images/agnhost:2.39
        args: [ "guestbook", "--backend-port", "6379" ]
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
        ports:
        - containerPort: 80

Jan 22 16:03:13.950: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/aa65bb68-9a6c-11ed-a697-56ea552f9d82/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-etcd-flatcar-k23-docker.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-357 create -f -'
Jan 22 16:03:14.278: INFO: stderr: ""
Jan 22 16:03:14.278: INFO: stdout: "deployment.apps/frontend created\n"
Jan 22 16:03:14.279: INFO: apiVersion: apps/v1
kind: Deployment
metadata:
  name: agnhost-primary
spec:
  replicas: 1
  selector:
    matchLabels:
      app: agnhost
      role: primary
      tier: backend
  template:
    metadata:
      labels:
        app: agnhost
        role: primary
        tier: backend
    spec:
      containers:
      - name: primary
        image: k8s.gcr.io/e2e-test-images/agnhost:2.39
        args: [ "guestbook", "--http-port", "6379" ]
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
        ports:
        - containerPort: 6379

Jan 22 16:03:14.279: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/aa65bb68-9a6c-11ed-a697-56ea552f9d82/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-etcd-flatcar-k23-docker.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-357 create -f -'
Jan 22 16:03:14.625: INFO: stderr: ""
Jan 22 16:03:14.625: INFO: stdout: "deployment.apps/agnhost-primary created\n"
Jan 22 16:03:14.625: INFO: apiVersion: apps/v1
kind: Deployment
metadata:
  name: agnhost-replica
spec:
  replicas: 2
  selector:
    matchLabels:
      app: agnhost
      role: replica
      tier: backend
  template:
    metadata:
      labels:
        app: agnhost
        role: replica
        tier: backend
    spec:
      containers:
      - name: replica
        image: k8s.gcr.io/e2e-test-images/agnhost:2.39
        args: [ "guestbook", "--replicaof", "agnhost-primary", "--http-port", "6379" ]
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
        ports:
        - containerPort: 6379

Jan 22 16:03:14.625: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/aa65bb68-9a6c-11ed-a697-56ea552f9d82/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-etcd-flatcar-k23-docker.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-357 create -f -'
Jan 22 16:03:14.986: INFO: stderr: ""
Jan 22 16:03:14.986: INFO: stdout: "deployment.apps/agnhost-replica created\n"
�[1mSTEP�[0m: validating guestbook app
Jan 22 16:03:14.986: INFO: Waiting for all frontend pods to be Running.
Jan 22 16:03:30.039: INFO: Waiting for frontend to serve content.
Jan 22 16:03:30.081: INFO: Trying to add a new entry to the guestbook.
Jan 22 16:03:35.121: INFO: Failed to get response from guestbook. err: the server responded with the status code 417 but did not return more information (get services frontend), response: 
Jan 22 16:03:45.157: INFO: Failed to get response from guestbook. err: the server responded with the status code 417 but did not return more information (get services frontend), response: 
Jan 22 16:03:55.197: INFO: Failed to get response from guestbook. err: the server responded with the status code 417 but did not return more information (get services frontend), response: 
Jan 22 16:04:05.236: INFO: Failed to get response from guestbook. err: the server responded with the status code 417 but did not return more information (get services frontend), response: 
Jan 22 16:04:15.271: INFO: Failed to get response from guestbook. err: the server responded with the status code 417 but did not return more information (get services frontend), response: 
Jan 22 16:04:25.305: INFO: Failed to get response from guestbook. err: the server responded with the status code 417 but did not return more information (get services frontend), response: 
Jan 22 16:04:35.342: INFO: Failed to get response from guestbook. err: the server responded with the status code 417 but did not return more information (get services frontend), response: 
Jan 22 16:04:45.385: INFO: Failed to get response from guestbook. err: the server responded with the status code 417 but did not return more information (get services frontend), response: 
Jan 22 16:04:55.426: INFO: Failed to get response from guestbook. err: the server responded with the status code 417 but did not return more information (get services frontend), response: 
Jan 22 16:05:05.461: INFO: Failed to get response from guestbook. err: the server responded with the status code 417 but did not return more information (get services frontend), response: 
Jan 22 16:05:15.499: INFO: Failed to get response from guestbook. err: the server responded with the status code 417 but did not return more information (get services frontend), response: 
Jan 22 16:05:25.533: INFO: Failed to get response from guestbook. err: the server responded with the status code 417 but did not return more information (get services frontend), response: 
Jan 22 16:05:35.568: INFO: Failed to get response from guestbook. err: the server responded with the status code 417 but did not return more information (get services frontend), response: 
Jan 22 16:05:45.615: INFO: Failed to get response from guestbook. err: the server responded with the status code 417 but did not return more information (get services frontend), response: 
Jan 22 16:05:55.650: INFO: Failed to get response from guestbook. err: the server responded with the status code 417 but did not return more information (get services frontend), response: 
Jan 22 16:06:05.686: INFO: Failed to get response from guestbook. err: the server responded with the status code 417 but did not return more information (get services frontend), response: 
Jan 22 16:06:15.725: INFO: Failed to get response from guestbook. err: the server responded with the status code 417 but did not return more information (get services frontend), response: 
Jan 22 16:06:25.771: INFO: Failed to get response from guestbook. err: the server responded with the status code 417 but did not return more information (get services frontend), response: 
Jan 22 16:06:30.772: FAIL: Cannot added new entry in 180 seconds.

Full Stack Trace
k8s.io/kubernetes/test/e2e/kubectl.glob..func1.7.2()
	/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl/kubectl.go:375 +0x147
k8s.io/kubernetes/test/e2e.RunE2ETests(0x24eec17?)
	_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:133 +0x6b7
k8s.io/kubernetes/test/e2e.TestE2E(0x24602d9?)
	_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:136 +0x19
testing.tRunner(0xc000d2e1a0, 0x72ecb90)
	/usr/local/go/src/testing/testing.go:1446 +0x10b
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:1493 +0x35f
�[1mSTEP�[0m: using delete to clean up resources
Jan 22 16:06:30.773: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/aa65bb68-9a6c-11ed-a697-56ea552f9d82/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-etcd-flatcar-k23-docker.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-357 delete --grace-period=0 --force -f -'
Jan 22 16:06:31.001: INFO: stderr: "warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.\n"
Jan 22 16:06:31.001: INFO: stdout: "service \"agnhost-replica\" force deleted\n"
�[1mSTEP�[0m: using delete to clean up resources
Jan 22 16:06:31.001: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/aa65bb68-9a6c-11ed-a697-56ea552f9d82/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-etcd-flatcar-k23-docker.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-357 delete --grace-period=0 --force -f -'
Jan 22 16:06:31.224: INFO: stderr: "warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.\n"
Jan 22 16:06:31.225: INFO: stdout: "service \"agnhost-primary\" force deleted\n"
�[1mSTEP�[0m: using delete to clean up resources
Jan 22 16:06:31.225: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/aa65bb68-9a6c-11ed-a697-56ea552f9d82/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-etcd-flatcar-k23-docker.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-357 delete --grace-period=0 --force -f -'
Jan 22 16:06:31.446: INFO: stderr: "warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.\n"
Jan 22 16:06:31.446: INFO: stdout: "service \"frontend\" force deleted\n"
�[1mSTEP�[0m: using delete to clean up resources
Jan 22 16:06:31.446: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/aa65bb68-9a6c-11ed-a697-56ea552f9d82/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-etcd-flatcar-k23-docker.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-357 delete --grace-period=0 --force -f -'
Jan 22 16:06:31.644: INFO: stderr: "warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.\n"
Jan 22 16:06:31.644: INFO: stdout: "deployment.apps \"frontend\" force deleted\n"
�[1mSTEP�[0m: using delete to clean up resources
Jan 22 16:06:31.645: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/aa65bb68-9a6c-11ed-a697-56ea552f9d82/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-etcd-flatcar-k23-docker.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-357 delete --grace-period=0 --force -f -'
Jan 22 16:06:31.852: INFO: stderr: "warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.\n"
Jan 22 16:06:31.852: INFO: stdout: "deployment.apps \"agnhost-primary\" force deleted\n"
�[1mSTEP�[0m: using delete to clean up resources
Jan 22 16:06:31.853: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/aa65bb68-9a6c-11ed-a697-56ea552f9d82/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-etcd-flatcar-k23-docker.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-357 delete --grace-period=0 --force -f -'
Jan 22 16:06:32.046: INFO: stderr: "warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.\n"
Jan 22 16:06:32.046: INFO: stdout: "deployment.apps \"agnhost-replica\" force deleted\n"
[AfterEach] [sig-cli] Kubectl client
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186
�[1mSTEP�[0m: Collecting events from namespace "kubectl-357".
�[1mSTEP�[0m: Found 39 events.
Jan 22 16:06:32.077: INFO: At 2023-01-22 16:03:14 +0000 UTC - event for agnhost-primary: {deployment-controller } ScalingReplicaSet: Scaled up replica set agnhost-primary-69cb998d54 to 1
Jan 22 16:06:32.077: INFO: At 2023-01-22 16:03:14 +0000 UTC - event for agnhost-primary-69cb998d54: {replicaset-controller } SuccessfulCreate: Created pod: agnhost-primary-69cb998d54-brrxv
Jan 22 16:06:32.077: INFO: At 2023-01-22 16:03:14 +0000 UTC - event for agnhost-primary-69cb998d54-brrxv: {default-scheduler } Scheduled: Successfully assigned kubectl-357/agnhost-primary-69cb998d54-brrxv to ip-172-20-39-104.us-east-2.compute.internal
Jan 22 16:06:32.077: INFO: At 2023-01-22 16:03:14 +0000 UTC - event for agnhost-replica: {deployment-controller } ScalingReplicaSet: Scaled up replica set agnhost-replica-6f5fb76474 to 2
Jan 22 16:06:32.077: INFO: At 2023-01-22 16:03:14 +0000 UTC - event for agnhost-replica-6f5fb76474: {replicaset-controller } SuccessfulCreate: Created pod: agnhost-replica-6f5fb76474-pmnld
Jan 22 16:06:32.077: INFO: At 2023-01-22 16:03:14 +0000 UTC - event for agnhost-replica-6f5fb76474: {replicaset-controller } SuccessfulCreate: Created pod: agnhost-replica-6f5fb76474-v7nq2
Jan 22 16:06:32.077: INFO: At 2023-01-22 16:03:14 +0000 UTC - event for agnhost-replica-6f5fb76474-pmnld: {default-scheduler } Scheduled: Successfully assigned kubectl-357/agnhost-replica-6f5fb76474-pmnld to ip-172-20-40-246.us-east-2.compute.internal
Jan 22 16:06:32.077: INFO: At 2023-01-22 16:03:14 +0000 UTC - event for agnhost-replica-6f5fb76474-v7nq2: {default-scheduler } Scheduled: Successfully assigned kubectl-357/agnhost-replica-6f5fb76474-v7nq2 to ip-172-20-45-186.us-east-2.compute.internal
Jan 22 16:06:32.077: INFO: At 2023-01-22 16:03:14 +0000 UTC - event for frontend: {deployment-controller } ScalingReplicaSet: Scaled up replica set frontend-5c4f744f96 to 3
Jan 22 16:06:32.077: INFO: At 2023-01-22 16:03:14 +0000 UTC - event for frontend-5c4f744f96: {replicaset-controller } SuccessfulCreate: Created pod: frontend-5c4f744f96-lbzp6
Jan 22 16:06:32.077: INFO: At 2023-01-22 16:03:14 +0000 UTC - event for frontend-5c4f744f96: {replicaset-controller } SuccessfulCreate: Created pod: frontend-5c4f744f96-zz7vx
Jan 22 16:06:32.077: INFO: At 2023-01-22 16:03:14 +0000 UTC - event for frontend-5c4f744f96: {replicaset-controller } SuccessfulCreate: Created pod: frontend-5c4f744f96-7tjxh
Jan 22 16:06:32.077: INFO: At 2023-01-22 16:03:14 +0000 UTC - event for frontend-5c4f744f96-7tjxh: {default-scheduler } Scheduled: Successfully assigned kubectl-357/frontend-5c4f744f96-7tjxh to ip-172-20-39-82.us-east-2.compute.internal
Jan 22 16:06:32.077: INFO: At 2023-01-22 16:03:14 +0000 UTC - event for frontend-5c4f744f96-lbzp6: {default-scheduler } Scheduled: Successfully assigned kubectl-357/frontend-5c4f744f96-lbzp6 to ip-172-20-45-186.us-east-2.compute.internal
Jan 22 16:06:32.077: INFO: At 2023-01-22 16:03:14 +0000 UTC - event for frontend-5c4f744f96-zz7vx: {default-scheduler } Scheduled: Successfully assigned kubectl-357/frontend-5c4f744f96-zz7vx to ip-172-20-40-246.us-east-2.compute.internal
Jan 22 16:06:32.077: INFO: At 2023-01-22 16:03:17 +0000 UTC - event for frontend-5c4f744f96-lbzp6: {kubelet ip-172-20-45-186.us-east-2.compute.internal} Created: Created container guestbook-frontend
Jan 22 16:06:32.077: INFO: At 2023-01-22 16:03:17 +0000 UTC - event for frontend-5c4f744f96-lbzp6: {kubelet ip-172-20-45-186.us-east-2.compute.internal} Pulled: Container image "k8s.gcr.io/e2e-test-images/agnhost:2.39" already present on machine
Jan 22 16:06:32.077: INFO: At 2023-01-22 16:03:17 +0000 UTC - event for frontend-5c4f744f96-lbzp6: {kubelet ip-172-20-45-186.us-east-2.compute.internal} Started: Started container guestbook-frontend
Jan 22 16:06:32.077: INFO: At 2023-01-22 16:03:21 +0000 UTC - event for agnhost-replica-6f5fb76474-v7nq2: {kubelet ip-172-20-45-186.us-east-2.compute.internal} Started: Started container replica
Jan 22 16:06:32.077: INFO: At 2023-01-22 16:03:21 +0000 UTC - event for agnhost-replica-6f5fb76474-v7nq2: {kubelet ip-172-20-45-186.us-east-2.compute.internal} Created: Created container replica
Jan 22 16:06:32.077: INFO: At 2023-01-22 16:03:21 +0000 UTC - event for agnhost-replica-6f5fb76474-v7nq2: {kubelet ip-172-20-45-186.us-east-2.compute.internal} Pulled: Container image "k8s.gcr.io/e2e-test-images/agnhost:2.39" already present on machine
Jan 22 16:06:32.077: INFO: At 2023-01-22 16:03:24 +0000 UTC - event for frontend-5c4f744f96-7tjxh: {kubelet ip-172-20-39-82.us-east-2.compute.internal} Pulled: Container image "k8s.gcr.io/e2e-test-images/agnhost:2.39" already present on machine
Jan 22 16:06:32.077: INFO: At 2023-01-22 16:03:24 +0000 UTC - event for frontend-5c4f744f96-7tjxh: {kubelet ip-172-20-39-82.us-east-2.compute.internal} Created: Created container guestbook-frontend
Jan 22 16:06:32.077: INFO: At 2023-01-22 16:03:25 +0000 UTC - event for frontend-5c4f744f96-7tjxh: {kubelet ip-172-20-39-82.us-east-2.compute.internal} Started: Started container guestbook-frontend
Jan 22 16:06:32.077: INFO: At 2023-01-22 16:03:28 +0000 UTC - event for frontend-5c4f744f96-zz7vx: {kubelet ip-172-20-40-246.us-east-2.compute.internal} Started: Started container guestbook-frontend
Jan 22 16:06:32.077: INFO: At 2023-01-22 16:03:28 +0000 UTC - event for frontend-5c4f744f96-zz7vx: {kubelet ip-172-20-40-246.us-east-2.compute.internal} Created: Created container guestbook-frontend
Jan 22 16:06:32.077: INFO: At 2023-01-22 16:03:28 +0000 UTC - event for frontend-5c4f744f96-zz7vx: {kubelet ip-172-20-40-246.us-east-2.compute.internal} Pulled: Container image "k8s.gcr.io/e2e-test-images/agnhost:2.39" already present on machine
Jan 22 16:06:32.077: INFO: At 2023-01-22 16:03:30 +0000 UTC - event for agnhost-replica-6f5fb76474-pmnld: {kubelet ip-172-20-40-246.us-east-2.compute.internal} Pulled: Container image "k8s.gcr.io/e2e-test-images/agnhost:2.39" already present on machine
Jan 22 16:06:32.077: INFO: At 2023-01-22 16:03:30 +0000 UTC - event for agnhost-replica-6f5fb76474-pmnld: {kubelet ip-172-20-40-246.us-east-2.compute.internal} Created: Created container replica
Jan 22 16:06:32.077: INFO: At 2023-01-22 16:03:31 +0000 UTC - event for agnhost-replica-6f5fb76474-pmnld: {kubelet ip-172-20-40-246.us-east-2.compute.internal} Started: Started container replica
Jan 22 16:06:32.077: INFO: At 2023-01-22 16:03:58 +0000 UTC - event for agnhost-replica-6f5fb76474-v7nq2: {kubelet ip-172-20-45-186.us-east-2.compute.internal} BackOff: Back-off restarting failed container
Jan 22 16:06:32.077: INFO: At 2023-01-22 16:04:08 +0000 UTC - event for agnhost-replica-6f5fb76474-pmnld: {kubelet ip-172-20-40-246.us-east-2.compute.internal} BackOff: Back-off restarting failed container
Jan 22 16:06:32.077: INFO: At 2023-01-22 16:04:15 +0000 UTC - event for agnhost-primary-69cb998d54-brrxv: {kubelet ip-172-20-39-104.us-east-2.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "abf00191739b03c11f3809292870808bb68578454f647e870ee07c2306784ee4" network for pod "agnhost-primary-69cb998d54-brrxv": networkPlugin cni failed to set up pod "agnhost-primary-69cb998d54-brrxv_kubectl-357" network: unable to connect to Cilium daemon: failed to create cilium agent client after 30.000000 seconds timeout: Get "http:///var/run/cilium/cilium.sock/v1/config": dial unix /var/run/cilium/cilium.sock: connect: no such file or directory
Is the agent running?
Jan 22 16:06:32.077: INFO: At 2023-01-22 16:04:20 +0000 UTC - event for agnhost-primary-69cb998d54-brrxv: {kubelet ip-172-20-39-104.us-east-2.compute.internal} SandboxChanged: Pod sandbox changed, it will be killed and re-created.
Jan 22 16:06:32.077: INFO: At 2023-01-22 16:05:51 +0000 UTC - event for agnhost-primary-69cb998d54-brrxv: {kubelet ip-172-20-39-104.us-east-2.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "fe1faafec1054b79e6e42f99ada06c74d43b9beb89eff5784510ff8274c9ecbb" network for pod "agnhost-primary-69cb998d54-brrxv": networkPlugin cni failed to set up pod "agnhost-primary-69cb998d54-brrxv_kubectl-357" network: unable to connect to Cilium daemon: failed to create cilium agent client after 30.000000 seconds timeout: Get "http:///var/run/cilium/cilium.sock/v1/config": dial unix /var/run/cilium/cilium.sock: connect: no such file or directory
Is the agent running?
Jan 22 16:06:32.077: INFO: At 2023-01-22 16:06:21 +0000 UTC - event for agnhost-primary-69cb998d54-brrxv: {kubelet ip-172-20-39-104.us-east-2.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "951398fcadc4c264efccb60b1d29b2858aa518cd1575e516ebcd601408353c43" network for pod "agnhost-primary-69cb998d54-brrxv": networkPlugin cni failed to set up pod "agnhost-primary-69cb998d54-brrxv_kubectl-357" network: Unable to create endpoint: [PUT /endpoint/{id}][429] putEndpointIdTooManyRequests 
Jan 22 16:06:32.077: INFO: At 2023-01-22 16:06:31 +0000 UTC - event for frontend-5c4f744f96-7tjxh: {kubelet ip-172-20-39-82.us-east-2.compute.internal} Killing: Stopping container guestbook-frontend
Jan 22 16:06:32.077: INFO: At 2023-01-22 16:06:31 +0000 UTC - event for frontend-5c4f744f96-lbzp6: {kubelet ip-172-20-45-186.us-east-2.compute.internal} Killing: Stopping container guestbook-frontend
Jan 22 16:06:32.077: INFO: At 2023-01-22 16:06:31 +0000 UTC - event for frontend-5c4f744f96-zz7vx: {kubelet ip-172-20-40-246.us-east-2.compute.internal} Killing: Stopping container guestbook-frontend
Jan 22 16:06:32.108: INFO: POD                               NODE                                         PHASE    GRACE  CONDITIONS
Jan 22 16:06:32.108: INFO: agnhost-primary-69cb998d54-brrxv  ip-172-20-39-104.us-east-2.compute.internal  Pending  30s    [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-22 16:03:14 +0000 UTC  } {Ready False 0001-01-01 00:00:00 +0000 UTC 2023-01-22 16:03:14 +0000 UTC ContainersNotReady containers with unready status: [primary]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2023-01-22 16:03:14 +0000 UTC ContainersNotReady containers with unready status: [primary]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-22 16:03:14 +0000 UTC  }]
Jan 22 16:06:32.108: INFO: agnhost-replica-6f5fb76474-pmnld  ip-172-20-40-246.us-east-2.compute.internal  Running  30s    [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-22 16:03:14 +0000 UTC  } {Ready False 0001-01-01 00:00:00 +0000 UTC 2023-01-22 16:06:24 +0000 UTC ContainersNotReady containers with unready status: [replica]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2023-01-22 16:06:24 +0000 UTC ContainersNotReady containers with unready status: [replica]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-22 16:03:14 +0000 UTC  }]
Jan 22 16:06:32.108: INFO: agnhost-replica-6f5fb76474-v7nq2  ip-172-20-45-186.us-east-2.compute.internal  Running  30s    [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-22 16:03:15 +0000 UTC  } {Ready False 0001-01-01 00:00:00 +0000 UTC 2023-01-22 16:06:21 +0000 UTC ContainersNotReady containers with unready status: [replica]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2023-01-22 16:06:21 +0000 UTC ContainersNotReady containers with unready status: [replica]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-22 16:03:14 +0000 UTC  }]
Jan 22 16:06:32.108: INFO: frontend-5c4f744f96-7tjxh         ip-172-20-39-82.us-east-2.compute.internal   Running  30s    [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-22 16:03:14 +0000 UTC  } {Ready True 0001-01-01 00:00:00 +0000 UTC 2023-01-22 16:03:25 +0000 UTC  } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-01-22 16:03:25 +0000 UTC  } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-22 16:03:14 +0000 UTC  }]
Jan 22 16:06:32.108: INFO: frontend-5c4f744f96-lbzp6         ip-172-20-45-186.us-east-2.compute.internal  Running  30s    [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-22 16:03:14 +0000 UTC  } {Ready True 0001-01-01 00:00:00 +0000 UTC 2023-01-22 16:03:17 +0000 UTC  } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-01-22 16:03:17 +0000 UTC  } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-22 16:03:14 +0000 UTC  }]
Jan 22 16:06:32.108: INFO: frontend-5c4f744f96-zz7vx         ip-172-20-40-246.us-east-2.compute.internal  Running  30s    [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-22 16:03:14 +0000 UTC  } {Ready True 0001-01-01 00:00:00 +0000 UTC 2023-01-22 16:03:29 +0000 UTC  } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-01-22 16:03:29 +0000 UTC  } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-22 16:03:14 +0000 UTC  }]
Jan 22 16:06:32.108: INFO: 
Jan 22 16:06:32.143: INFO: Unable to fetch kubectl-357/agnhost-primary-69cb998d54-brrxv/primary logs: the server rejected our request for an unknown reason (get pods agnhost-primary-69cb998d54-brrxv)
Jan 22 16:06:32.355: INFO: Unable to fetch kubectl-357/frontend-5c4f744f96-7tjxh/guestbook-frontend logs: the server could not find the requested resource (get pods frontend-5c4f744f96-7tjxh)
Jan 22 16:06:32.429: INFO: 
Logging node info for node ip-172-20-39-104.us-east-2.compute.internal
Jan 22 16:06:32.459: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-39-104.us-east-2.compute.internal    94a127fa-2d03-4fae-878f-95cc778a7072 10036 0 2023-01-22 15:57:23 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-east-2 failure-domain.beta.kubernetes.io/zone:us-east-2a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-39-104.us-east-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-east-2a topology.hostpath.csi/node:ip-172-20-39-104.us-east-2.compute.internal topology.kubernetes.io/region:us-east-2 topology.kubernetes.io/zone:us-east-2a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-446":"ip-172-20-39-104.us-east-2.compute.internal","ebs.csi.aws.com":"i-058f72b2ec21c63a8"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kops-controller Update v1 2023-01-22 15:57:23 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-22 15:57:23 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {kubelet Update v1 2023-01-22 15:57:23 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2023-01-22 16:02:09 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-22 16:02:16 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status} {e2e.test Update v1 2023-01-22 16:02:30 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} }]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///us-east-2a/i-058f72b2ec21c63a8,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{47441653760 0} {<nil>} 46329740Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4054806528 0} {<nil>} 3959772Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{42697488314 0} {<nil>} 42697488314 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3949948928 0} {<nil>} 3857372Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-22 16:06:27 +0000 UTC,LastTransitionTime:2023-01-22 15:57:23 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-22 16:06:27 +0000 UTC,LastTransitionTime:2023-01-22 15:57:23 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-22 16:06:27 +0000 UTC,LastTransitionTime:2023-01-22 15:57:23 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-22 16:06:27 +0000 UTC,LastTransitionTime:2023-01-22 15:57:44 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.39.104,},NodeAddress{Type:ExternalIP,Address:3.133.79.156,},NodeAddress{Type:Hostname,Address:ip-172-20-39-104.us-east-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-39-104.us-east-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-133-79-156.us-east-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2455e84fc0e92553e5c201c3d73752,SystemUUID:ec2455e8-4fc0-e925-53e5-c201c3d73752,BootID:baa49577-2567-45b4-b63f-1f65be12643c,KernelVersion:5.15.86-flatcar,OSImage:Flatcar Container Linux by Kinvolk 3446.1.0 (Oklo),ContainerRuntimeVersion:docker://20.10.21,KubeletVersion:v1.23.16,KubeProxyVersion:v1.23.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456551323,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126853136,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:99c0d6f1ad24a1aa1905d9c6534d193f268f7b23f9add2ae6bb41f31094bdd5c k8s.gcr.io/e2e-test-images/nautilus:1.5],SizeBytes:124738681,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123774598,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.16 registry.k8s.io/kube-proxy-amd64:v1.23.16],SizeBytes:110829687,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:94469968,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56363051,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53940090,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53515772,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53483352,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:50249337,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:47903915,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29972082,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19575418,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18683218,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17605286,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17159270,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16028126,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-446^2962d889-9a6e-11ed-aca1-b240603228b3 kubernetes.io/csi/ebs.csi.aws.com^vol-00f9ca3f8a10b690b],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-00f9ca3f8a10b690b,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-446^2962d889-9a6e-11ed-aca1-b240603228b3,DevicePath:,},},Config:nil,},}
Jan 22 16:06:32.459: INFO: 
Logging kubelet events for node ip-172-20-39-104.us-east-2.compute.internal
Jan 22 16:06:32.491: INFO: 
Logging pods the kubelet thinks is on node ip-172-20-39-104.us-east-2.compute.internal
Jan 22 16:06:32.569: INFO: cilium-cl25d started at 2023-01-22 15:57:24 +0000 UTC (1+1 container statuses recorded)
Jan 22 16:06:32.569: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Jan 22 16:06:32.569: INFO: 	Container cilium-agent ready: true, restart count 2
Jan 22 16:06:32.569: INFO: pod-fdd8c0f7-f69b-4a5a-9c0f-501002697793 started at 2023-01-22 16:03:46 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:32.569: INFO: 	Container write-pod ready: false, restart count 0
Jan 22 16:06:32.569: INFO: netserver-0 started at 2023-01-22 16:03:48 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:32.569: INFO: 	Container webserver ready: false, restart count 0
Jan 22 16:06:32.569: INFO: test-rolling-update-with-lb-7dcdf997cd-h69x9 started at 2023-01-22 16:04:41 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:32.569: INFO: 	Container agnhost ready: false, restart count 0
Jan 22 16:06:32.569: INFO: hostpath-symlink-prep-provisioning-7487 started at 2023-01-22 16:04:50 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:32.569: INFO: 	Container init-volume-provisioning-7487 ready: false, restart count 0
Jan 22 16:06:32.569: INFO: pod-subpath-test-preprovisionedpv-thmc started at 2023-01-22 16:05:22 +0000 UTC (1+1 container statuses recorded)
Jan 22 16:06:32.569: INFO: 	Init container init-volume-preprovisionedpv-thmc ready: false, restart count 0
Jan 22 16:06:32.569: INFO: 	Container test-container-subpath-preprovisionedpv-thmc ready: false, restart count 0
Jan 22 16:06:32.569: INFO: ebs-csi-node-8fr2s started at 2023-01-22 15:57:24 +0000 UTC (0+3 container statuses recorded)
Jan 22 16:06:32.569: INFO: 	Container ebs-plugin ready: true, restart count 0
Jan 22 16:06:32.569: INFO: 	Container liveness-probe ready: true, restart count 0
Jan 22 16:06:32.569: INFO: 	Container node-driver-registrar ready: true, restart count 0
Jan 22 16:06:32.569: INFO: pod-d0184f4b-c93e-456b-bc7d-3eaf544b4775 started at 2023-01-22 16:03:21 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:32.569: INFO: 	Container write-pod ready: false, restart count 0
Jan 22 16:06:32.569: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-265ss started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:32.569: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: false, restart count 0
Jan 22 16:06:32.569: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-n866x started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:32.569: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: false, restart count 0
Jan 22 16:06:32.569: INFO: csi-mockplugin-attacher-0 started at 2023-01-22 16:02:37 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:32.569: INFO: 	Container csi-attacher ready: false, restart count 0
Jan 22 16:06:32.569: INFO: hostexec-ip-172-20-39-104.us-east-2.compute.internal-x7v8d started at 2023-01-22 16:03:43 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:32.569: INFO: 	Container agnhost-container ready: true, restart count 0
Jan 22 16:06:32.569: INFO: csi-mockplugin-resizer-0 started at 2023-01-22 16:02:37 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:32.569: INFO: 	Container csi-resizer ready: false, restart count 0
Jan 22 16:06:32.569: INFO: csi-mockplugin-attacher-0 started at 2023-01-22 16:03:27 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:32.569: INFO: 	Container csi-attacher ready: false, restart count 0
Jan 22 16:06:32.569: INFO: busybox-c1850765-faac-4c06-871f-90efbaeb9304 started at 2023-01-22 16:04:46 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:32.569: INFO: 	Container busybox ready: false, restart count 0
Jan 22 16:06:32.569: INFO: test-pod-51789469-aa8b-4bbe-a687-5f045bfe8f3f started at 2023-01-22 16:05:55 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:32.569: INFO: 	Container agnhost-container ready: false, restart count 0
Jan 22 16:06:32.569: INFO: coredns-867df8f45c-7d727 started at 2023-01-22 15:57:44 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:32.569: INFO: 	Container coredns ready: true, restart count 0
Jan 22 16:06:32.569: INFO: test-rolling-update-with-lb-79bf9f6988-6rvkk started at 2023-01-22 16:01:39 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:32.569: INFO: 	Container agnhost ready: true, restart count 0
Jan 22 16:06:32.569: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-b44bm started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:32.569: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: false, restart count 0
Jan 22 16:06:32.569: INFO: pod-995c444c-9bc9-46ca-9def-361673d2589d started at 2023-01-22 16:03:45 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:32.569: INFO: 	Container test-container ready: false, restart count 0
Jan 22 16:06:32.569: INFO: pfpod started at 2023-01-22 16:04:22 +0000 UTC (0+2 container statuses recorded)
Jan 22 16:06:32.569: INFO: 	Container portforwardtester ready: false, restart count 0
Jan 22 16:06:32.569: INFO: 	Container readiness ready: false, restart count 0
Jan 22 16:06:32.569: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-zltrj started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:32.569: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: false, restart count 0
Jan 22 16:06:32.569: INFO: agnhost-primary-69cb998d54-brrxv started at 2023-01-22 16:03:14 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:32.569: INFO: 	Container primary ready: false, restart count 0
Jan 22 16:06:32.569: INFO: ss-1 started at 2023-01-22 16:02:19 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:32.569: INFO: 	Container webserver ready: false, restart count 0
Jan 22 16:06:32.569: INFO: busybox-6a91c0d9-7f05-4dcb-ba51-63e51c292fca started at 2023-01-22 16:04:28 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:32.569: INFO: 	Container busybox ready: true, restart count 0
Jan 22 16:06:32.569: INFO: pod-subpath-test-inlinevolume-r9xz started at 2023-01-22 16:02:21 +0000 UTC (2+1 container statuses recorded)
Jan 22 16:06:32.569: INFO: 	Init container init-volume-inlinevolume-r9xz ready: false, restart count 0
Jan 22 16:06:32.569: INFO: 	Init container test-init-volume-inlinevolume-r9xz ready: false, restart count 0
Jan 22 16:06:32.569: INFO: 	Container test-container-subpath-inlinevolume-r9xz ready: false, restart count 0
Jan 22 16:06:32.569: INFO: inline-volume-tester-jpszl started at 2023-01-22 16:02:25 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:32.570: INFO: 	Container csi-volume-tester ready: true, restart count 0
Jan 22 16:06:32.570: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-zkthv started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:32.570: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: false, restart count 0
Jan 22 16:06:32.570: INFO: coredns-autoscaler-557ccb4c66-44k6v started at 2023-01-22 15:57:44 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:32.570: INFO: 	Container autoscaler ready: true, restart count 0
Jan 22 16:06:32.570: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-ltqph started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:32.570: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: false, restart count 0
Jan 22 16:06:32.570: INFO: csi-mockplugin-0 started at 2023-01-22 16:02:37 +0000 UTC (0+3 container statuses recorded)
Jan 22 16:06:32.570: INFO: 	Container csi-provisioner ready: false, restart count 0
Jan 22 16:06:32.570: INFO: 	Container driver-registrar ready: false, restart count 0
Jan 22 16:06:32.570: INFO: 	Container mock ready: false, restart count 0
Jan 22 16:06:32.570: INFO: hostexec-ip-172-20-39-104.us-east-2.compute.internal-5d8rz started at 2023-01-22 16:03:10 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:32.570: INFO: 	Container agnhost-container ready: true, restart count 0
Jan 22 16:06:32.570: INFO: netserver-0 started at 2023-01-22 16:03:41 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:32.570: INFO: 	Container webserver ready: false, restart count 0
Jan 22 16:06:32.570: INFO: netserver-0 started at 2023-01-22 16:02:31 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:32.570: INFO: 	Container webserver ready: false, restart count 0
Jan 22 16:06:32.570: INFO: hostexec-ip-172-20-39-104.us-east-2.compute.internal-wmg7l started at 2023-01-22 16:05:07 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:32.570: INFO: 	Container agnhost-container ready: true, restart count 0
Jan 22 16:06:32.570: INFO: csi-mockplugin-0 started at 2023-01-22 16:03:27 +0000 UTC (0+3 container statuses recorded)
Jan 22 16:06:32.570: INFO: 	Container csi-provisioner ready: false, restart count 0
Jan 22 16:06:32.570: INFO: 	Container driver-registrar ready: false, restart count 0
Jan 22 16:06:32.570: INFO: 	Container mock ready: false, restart count 0
Jan 22 16:06:32.570: INFO: csi-hostpathplugin-0 started at 2023-01-22 16:02:09 +0000 UTC (0+7 container statuses recorded)
Jan 22 16:06:32.570: INFO: 	Container csi-attacher ready: true, restart count 0
Jan 22 16:06:32.570: INFO: 	Container csi-provisioner ready: true, restart count 0
Jan 22 16:06:32.570: INFO: 	Container csi-resizer ready: true, restart count 0
Jan 22 16:06:32.570: INFO: 	Container csi-snapshotter ready: true, restart count 0
Jan 22 16:06:32.570: INFO: 	Container hostpath ready: true, restart count 0
Jan 22 16:06:32.570: INFO: 	Container liveness-probe ready: true, restart count 0
Jan 22 16:06:32.570: INFO: 	Container node-driver-registrar ready: true, restart count 0
Jan 22 16:06:32.570: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-mn2wn started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:32.570: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: false, restart count 0
Jan 22 16:06:32.570: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-dmxrv started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:32.570: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: false, restart count 0
Jan 22 16:06:32.570: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-7sknq started at 2023-01-22 16:02:31 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:32.570: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: false, restart count 0
Jan 22 16:06:32.570: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-hrlf4 started at 2023-01-22 16:02:31 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:32.570: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: false, restart count 0
Jan 22 16:06:32.909: INFO: 
Latency metrics for node ip-172-20-39-104.us-east-2.compute.internal
Jan 22 16:06:32.909: INFO: 
Logging node info for node ip-172-20-39-82.us-east-2.compute.internal
Jan 22 16:06:32.940: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-39-82.us-east-2.compute.internal    fab1195f-32aa-4044-ac45-046bbfde063a 8883 0 2023-01-22 15:57:34 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-east-2 failure-domain.beta.kubernetes.io/zone:us-east-2a io.kubernetes.storage.mock/node:some-mock-node kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-39-82.us-east-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-east-2a topology.kubernetes.io/region:us-east-2 topology.kubernetes.io/zone:us-east-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0a00ce72536f23489"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kops-controller Update v1 2023-01-22 15:57:34 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-22 15:57:34 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2023-01-22 15:57:34 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2023-01-22 16:01:40 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {e2e.test Update v1 2023-01-22 16:02:30 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kubelet Update v1 2023-01-22 16:03:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///us-east-2a/i-0a00ce72536f23489,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{47441653760 0} {<nil>} 46329740Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4054806528 0} {<nil>} 3959772Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{42697488314 0} {<nil>} 42697488314 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3949948928 0} {<nil>} 3857372Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-22 16:04:12 +0000 UTC,LastTransitionTime:2023-01-22 15:57:34 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-22 16:04:12 +0000 UTC,LastTransitionTime:2023-01-22 15:57:34 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-22 16:04:12 +0000 UTC,LastTransitionTime:2023-01-22 15:57:34 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-22 16:04:12 +0000 UTC,LastTransitionTime:2023-01-22 15:57:54 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.39.82,},NodeAddress{Type:ExternalIP,Address:3.14.247.52,},NodeAddress{Type:Hostname,Address:ip-172-20-39-82.us-east-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-39-82.us-east-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-14-247-52.us-east-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bb4ac3588554994a702eabb6be703,SystemUUID:ec2bb4ac-3588-5549-94a7-02eabb6be703,BootID:82a481e4-e615-4371-bbc2-0b7f74e99966,KernelVersion:5.15.86-flatcar,OSImage:Flatcar Container Linux by Kinvolk 3446.1.0 (Oklo),ContainerRuntimeVersion:docker://20.10.21,KubeletVersion:v1.23.16,KubeProxyVersion:v1.23.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456551323,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:253346057,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126853136,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:99c0d6f1ad24a1aa1905d9c6534d193f268f7b23f9add2ae6bb41f31094bdd5c k8s.gcr.io/e2e-test-images/nautilus:1.5],SizeBytes:124738681,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123774598,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.16 registry.k8s.io/kube-proxy-amd64:v1.23.16],SizeBytes:110829687,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:94469968,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56366111,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53484608,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:50249337,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19646681,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19575418,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17664775,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17605286,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16028126,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-07558a7d8ec7e2f44],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-07558a7d8ec7e2f44,DevicePath:,},},Config:nil,},}
Jan 22 16:06:32.941: INFO: 
Logging kubelet events for node ip-172-20-39-82.us-east-2.compute.internal
Jan 22 16:06:32.981: INFO: 
Logging pods the kubelet thinks is on node ip-172-20-39-82.us-east-2.compute.internal
Jan 22 16:06:33.021: INFO: netserver-1 started at 2023-01-22 16:02:31 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.021: INFO: 	Container webserver ready: true, restart count 0
Jan 22 16:06:33.021: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-s2z7v started at 2023-01-22 16:02:31 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.021: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:06:33.021: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-k72n5 started at 2023-01-22 16:02:31 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.021: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:06:33.021: INFO: coredns-867df8f45c-7m6hf started at 2023-01-22 15:58:04 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.021: INFO: 	Container coredns ready: true, restart count 0
Jan 22 16:06:33.021: INFO: ss-0 started at 2023-01-22 16:01:47 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.021: INFO: 	Container webserver ready: true, restart count 0
Jan 22 16:06:33.021: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-cvg4m started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.021: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:06:33.021: INFO: netserver-1 started at 2023-01-22 16:03:48 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.021: INFO: 	Container webserver ready: true, restart count 0
Jan 22 16:06:33.021: INFO: test-rolling-update-with-lb-79bf9f6988-kwp9d started at 2023-01-22 16:01:39 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.021: INFO: 	Container agnhost ready: true, restart count 0
Jan 22 16:06:33.021: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-tbzwq started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.021: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:06:33.021: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-4pwz8 started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.021: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:06:33.021: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-f548n started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.021: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:06:33.021: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-dwhkt started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.021: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:06:33.021: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-m8pkb started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.021: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:06:33.021: INFO: netserver-1 started at 2023-01-22 16:03:41 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.021: INFO: 	Container webserver ready: true, restart count 0
Jan 22 16:06:33.021: INFO: cilium-dpwnn started at 2023-01-22 15:57:34 +0000 UTC (1+1 container statuses recorded)
Jan 22 16:06:33.021: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Jan 22 16:06:33.021: INFO: 	Container cilium-agent ready: true, restart count 2
Jan 22 16:06:33.021: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-hzsmm started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.021: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:06:33.021: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-pr6wn started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.021: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:06:33.021: INFO: deployment-shared-map-item-removal-7c658794b9-9hwvl started at 2023-01-22 16:05:25 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.021: INFO: 	Container nginx ready: false, restart count 0
Jan 22 16:06:33.021: INFO: bin-false8b2b933f-04ff-47e2-93dd-bb6998081ce8 started at 2023-01-22 16:06:18 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.021: INFO: 	Container bin-false8b2b933f-04ff-47e2-93dd-bb6998081ce8 ready: false, restart count 0
Jan 22 16:06:33.021: INFO: ebs-csi-node-5xvpn started at 2023-01-22 15:57:34 +0000 UTC (0+3 container statuses recorded)
Jan 22 16:06:33.021: INFO: 	Container ebs-plugin ready: true, restart count 0
Jan 22 16:06:33.021: INFO: 	Container liveness-probe ready: true, restart count 0
Jan 22 16:06:33.021: INFO: 	Container node-driver-registrar ready: true, restart count 0
Jan 22 16:06:33.145: INFO: 
Latency metrics for node ip-172-20-39-82.us-east-2.compute.internal
Jan 22 16:06:33.145: INFO: 
Logging node info for node ip-172-20-40-246.us-east-2.compute.internal
Jan 22 16:06:33.175: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-40-246.us-east-2.compute.internal    a4283492-f7a9-4ac6-aaab-f667bea07522 9744 0 2023-01-22 15:57:26 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-east-2 failure-domain.beta.kubernetes.io/zone:us-east-2a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-40-246.us-east-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-east-2a topology.hostpath.csi/node:ip-172-20-40-246.us-east-2.compute.internal topology.kubernetes.io/region:us-east-2 topology.kubernetes.io/zone:us-east-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0a5034e547859f6df"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kubelet Update v1 2023-01-22 15:57:26 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kops-controller Update v1 2023-01-22 15:57:27 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-22 15:57:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {e2e.test Update v1 2023-01-22 16:02:30 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kubelet Update v1 2023-01-22 16:05:06 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2023-01-22 16:05:16 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///us-east-2a/i-0a5034e547859f6df,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{47441653760 0} {<nil>} 46329740Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4054806528 0} {<nil>} 3959772Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{42697488314 0} {<nil>} 42697488314 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3949948928 0} {<nil>} 3857372Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-22 16:05:57 +0000 UTC,LastTransitionTime:2023-01-22 15:57:26 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-22 16:05:57 +0000 UTC,LastTransitionTime:2023-01-22 15:57:26 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-22 16:05:57 +0000 UTC,LastTransitionTime:2023-01-22 15:57:26 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-22 16:05:57 +0000 UTC,LastTransitionTime:2023-01-22 15:57:47 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.40.246,},NodeAddress{Type:ExternalIP,Address:3.22.242.15,},NodeAddress{Type:Hostname,Address:ip-172-20-40-246.us-east-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-40-246.us-east-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-22-242-15.us-east-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2ac3af6059e59b70f877f01430277f,SystemUUID:ec2ac3af-6059-e59b-70f8-77f01430277f,BootID:f03e4ca2-c137-40c8-ba1b-f79f8d612996,KernelVersion:5.15.86-flatcar,OSImage:Flatcar Container Linux by Kinvolk 3446.1.0 (Oklo),ContainerRuntimeVersion:docker://20.10.21,KubeletVersion:v1.23.16,KubeProxyVersion:v1.23.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456551323,},ContainerImage{Names:[nginx@sha256:b8f2383a95879e1ae064940d9a200f67a6c79e710ed82ac42263397367e7cc4e nginx:latest],SizeBytes:141788385,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126853136,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123774598,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.16 registry.k8s.io/kube-proxy-amd64:v1.23.16],SizeBytes:110829687,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:94469968,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56366111,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56363051,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53940090,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53515772,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53483352,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:b9e2958a3dd879e3cf11142228c6d073d0fc4ea2e857c3be6f4fb0ab5fb2c937 k8s.gcr.io/e2e-test-images/nonroot:1.2],SizeBytes:42320060,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29972082,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19646681,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19575418,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18683218,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17664775,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17605286,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17159270,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16028126,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-07be1c7978ad55547],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-07be1c7978ad55547,DevicePath:,},},Config:nil,},}
Jan 22 16:06:33.175: INFO: 
Logging kubelet events for node ip-172-20-40-246.us-east-2.compute.internal
Jan 22 16:06:33.207: INFO: 
Logging pods the kubelet thinks is on node ip-172-20-40-246.us-east-2.compute.internal
Jan 22 16:06:33.243: INFO: cilium-s627s started at 2023-01-22 15:57:27 +0000 UTC (1+1 container statuses recorded)
Jan 22 16:06:33.243: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Jan 22 16:06:33.243: INFO: 	Container cilium-agent ready: true, restart count 2
Jan 22 16:06:33.243: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-lrc95 started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.243: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:06:33.243: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-bgclt started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.243: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:06:33.243: INFO: test-rolling-update-with-lb-79bf9f6988-lvxhx started at 2023-01-22 16:01:39 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.243: INFO: 	Container agnhost ready: true, restart count 0
Jan 22 16:06:33.243: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-2kqq5 started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.243: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:06:33.243: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-jp49t started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.243: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:06:33.243: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-2xdb2 started at 2023-01-22 16:02:31 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.243: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:06:33.243: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-tg6vg started at 2023-01-22 16:02:31 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.243: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:06:33.243: INFO: netserver-2 started at 2023-01-22 16:03:49 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.243: INFO: 	Container webserver ready: true, restart count 0
Jan 22 16:06:33.243: INFO: ebs-csi-node-vrjbd started at 2023-01-22 15:57:27 +0000 UTC (0+3 container statuses recorded)
Jan 22 16:06:33.243: INFO: 	Container ebs-plugin ready: true, restart count 0
Jan 22 16:06:33.243: INFO: 	Container liveness-probe ready: true, restart count 0
Jan 22 16:06:33.244: INFO: 	Container node-driver-registrar ready: true, restart count 0
Jan 22 16:06:33.244: INFO: agnhost-replica-6f5fb76474-pmnld started at 2023-01-22 16:03:14 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.244: INFO: 	Container replica ready: false, restart count 4
Jan 22 16:06:33.244: INFO: netserver-2 started at 2023-01-22 16:02:31 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.244: INFO: 	Container webserver ready: true, restart count 0
Jan 22 16:06:33.244: INFO: hostexec-ip-172-20-40-246.us-east-2.compute.internal-kq2xw started at 2023-01-22 16:06:26 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.244: INFO: 	Container agnhost-container ready: true, restart count 0
Jan 22 16:06:33.244: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-26ll5 started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.244: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:06:33.244: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-2bwv6 started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.244: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:06:33.244: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-jp7lk started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.244: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:06:33.244: INFO: frontend-5c4f744f96-zz7vx started at 2023-01-22 16:03:14 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.244: INFO: 	Container guestbook-frontend ready: true, restart count 0
Jan 22 16:06:33.244: INFO: webserver-pod started at 2023-01-22 16:01:22 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.244: INFO: 	Container agnhost ready: false, restart count 0
Jan 22 16:06:33.244: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-pdmh6 started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.244: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:06:33.244: INFO: netserver-2 started at 2023-01-22 16:03:41 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.244: INFO: 	Container webserver ready: true, restart count 0
Jan 22 16:06:33.364: INFO: 
Latency metrics for node ip-172-20-40-246.us-east-2.compute.internal
Jan 22 16:06:33.364: INFO: 
Logging node info for node ip-172-20-45-186.us-east-2.compute.internal
Jan 22 16:06:33.395: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-45-186.us-east-2.compute.internal    d0ea7688-ae6f-4b0b-b4c0-87950ac02237 9896 0 2023-01-22 15:57:24 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-east-2 failure-domain.beta.kubernetes.io/zone:us-east-2a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-45-186.us-east-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-east-2a topology.hostpath.csi/node:ip-172-20-45-186.us-east-2.compute.internal topology.kubernetes.io/region:us-east-2 topology.kubernetes.io/zone:us-east-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0ce264dea7066c689"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kops-controller Update v1 2023-01-22 15:57:24 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-01-22 15:57:24 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2023-01-22 15:57:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {e2e.test Update v1 2023-01-22 16:02:30 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kubelet Update v1 2023-01-22 16:06:16 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///us-east-2a/i-0ce264dea7066c689,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{47441653760 0} {<nil>} 46329740Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4054806528 0} {<nil>} 3959772Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{42697488314 0} {<nil>} 42697488314 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3949948928 0} {<nil>} 3857372Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-22 16:06:16 +0000 UTC,LastTransitionTime:2023-01-22 15:57:24 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-22 16:06:16 +0000 UTC,LastTransitionTime:2023-01-22 15:57:24 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-22 16:06:16 +0000 UTC,LastTransitionTime:2023-01-22 15:57:24 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-22 16:06:16 +0000 UTC,LastTransitionTime:2023-01-22 15:57:55 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.45.186,},NodeAddress{Type:ExternalIP,Address:18.118.166.212,},NodeAddress{Type:Hostname,Address:ip-172-20-45-186.us-east-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-45-186.us-east-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-118-166-212.us-east-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2308cb46db6b634f318d31f03adcbc,SystemUUID:ec2308cb-46db-6b63-4f31-8d31f03adcbc,BootID:c1380477-0aee-480e-8dd1-20b7d9813f9d,KernelVersion:5.15.86-flatcar,OSImage:Flatcar Container Linux by Kinvolk 3446.1.0 (Oklo),ContainerRuntimeVersion:docker://20.10.21,KubeletVersion:v1.23.16,KubeProxyVersion:v1.23.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456551323,},ContainerImage{Names:[nginx@sha256:b8f2383a95879e1ae064940d9a200f67a6c79e710ed82ac42263397367e7cc4e nginx:latest],SizeBytes:141788385,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126853136,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123774598,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.16 registry.k8s.io/kube-proxy-amd64:v1.23.16],SizeBytes:110829687,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:94469968,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56366111,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56363051,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53940090,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53515772,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53484608,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53483352,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:49160266,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29972082,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19646681,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19575418,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18683218,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17664775,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17605286,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17159270,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16028126,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:4202236,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Jan 22 16:06:33.395: INFO: 
Logging kubelet events for node ip-172-20-45-186.us-east-2.compute.internal
Jan 22 16:06:33.427: INFO: 
Logging pods the kubelet thinks is on node ip-172-20-45-186.us-east-2.compute.internal
Jan 22 16:06:33.468: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-7f88x started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.468: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:06:33.468: INFO: cilium-zt6nc started at 2023-01-22 15:57:25 +0000 UTC (1+1 container statuses recorded)
Jan 22 16:06:33.469: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Jan 22 16:06:33.469: INFO: 	Container cilium-agent ready: true, restart count 2
Jan 22 16:06:33.469: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-8d2c7 started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.469: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:06:33.469: INFO: netserver-3 started at 2023-01-22 16:02:31 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.469: INFO: 	Container webserver ready: true, restart count 0
Jan 22 16:06:33.469: INFO: netserver-3 started at 2023-01-22 16:03:41 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.469: INFO: 	Container webserver ready: true, restart count 0
Jan 22 16:06:33.469: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-kxcmp started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.469: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:06:33.469: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-26q8s started at 2023-01-22 16:02:31 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.469: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:06:33.469: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-bzv42 started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.469: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:06:33.469: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-8q6b2 started at 2023-01-22 16:02:31 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.469: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:06:33.469: INFO: netserver-3 started at 2023-01-22 16:03:49 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.469: INFO: 	Container webserver ready: true, restart count 0
Jan 22 16:06:33.469: INFO: ebs-csi-node-45fpb started at 2023-01-22 15:57:25 +0000 UTC (0+3 container statuses recorded)
Jan 22 16:06:33.469: INFO: 	Container ebs-plugin ready: true, restart count 0
Jan 22 16:06:33.469: INFO: 	Container liveness-probe ready: true, restart count 0
Jan 22 16:06:33.469: INFO: 	Container node-driver-registrar ready: true, restart count 0
Jan 22 16:06:33.469: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-gmj8t started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.469: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:06:33.469: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-qjt7c started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.469: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:06:33.469: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-fq5q8 started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.469: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:06:33.469: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-7xqsc started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.469: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:06:33.598: INFO: 
Latency metrics for node ip-172-20-45-186.us-east-2.compute.internal
Jan 22 16:06:33.598: INFO: 
Logging node info for node ip-172-20-52-12.us-east-2.compute.internal
Jan 22 16:06:33.628: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-52-12.us-east-2.compute.internal    33149587-e10e-48d5-abaf-ca8103e85c14 6683 0 2023-01-22 15:56:13 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-east-2 failure-domain.beta.kubernetes.io/zone:us-east-2a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-52-12.us-east-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/control-plane: node-role.kubernetes.io/master: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-east-2a topology.kubernetes.io/region:us-east-2 topology.kubernetes.io/zone:us-east-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0f06483805887653c"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kubelet Update v1 2023-01-22 15:56:13 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {protokube Update v1 2023-01-22 15:56:32 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kops-controller Update v1 2023-01-22 15:56:36 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/master":{}}}} } {kube-controller-manager Update v1 2023-01-22 15:56:54 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-01-22 15:57:53 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///us-east-2a/i-0f06483805887653c,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{47441653760 0} {<nil>} 46329740Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3895427072 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{42697488314 0} {<nil>} 42697488314 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3790569472 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-22 16:03:11 +0000 UTC,LastTransitionTime:2023-01-22 15:56:13 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-22 16:03:11 +0000 UTC,LastTransitionTime:2023-01-22 15:56:13 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-22 16:03:11 +0000 UTC,LastTransitionTime:2023-01-22 15:56:13 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-22 16:03:11 +0000 UTC,LastTransitionTime:2023-01-22 15:56:54 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.52.12,},NodeAddress{Type:ExternalIP,Address:3.16.137.247,},NodeAddress{Type:Hostname,Address:ip-172-20-52-12.us-east-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-52-12.us-east-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-16-137-247.us-east-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec221e40526c51c23560b6cf49b17a65,SystemUUID:ec221e40-526c-51c2-3560-b6cf49b17a65,BootID:89f5d760-653a-4491-8f94-5adaf8195cda,KernelVersion:5.15.86-flatcar,OSImage:Flatcar Container Linux by Kinvolk 3446.1.0 (Oklo),ContainerRuntimeVersion:docker://20.10.21,KubeletVersion:v1.23.16,KubeProxyVersion:v1.23.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:66a453db625abb268f4b3bbefc5a34a171d81e6e8796cecca54cfd71775c77c4 registry.k8s.io/etcdadm/etcd-manager:v3.0.20221209],SizeBytes:662196755,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456551323,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64:v1.23.16 registry.k8s.io/kube-apiserver-amd64:v1.23.16],SizeBytes:129993155,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64:v1.23.16 registry.k8s.io/kube-controller-manager-amd64:v1.23.16],SizeBytes:119933673,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.16 registry.k8s.io/kube-proxy-amd64:v1.23.16],SizeBytes:110829687,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:110016956,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:105636625,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:102694452,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:94469968,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:57729376,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:55458084,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:54841901,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64:v1.23.16 registry.k8s.io/kube-scheduler-amd64:v1.23.16],SizeBytes:51845852,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19575418,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17605286,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:8785817,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Jan 22 16:06:33.628: INFO: 
Logging kubelet events for node ip-172-20-52-12.us-east-2.compute.internal
Jan 22 16:06:33.660: INFO: 
Logging pods the kubelet thinks is on node ip-172-20-52-12.us-east-2.compute.internal
Jan 22 16:06:33.700: INFO: kube-controller-manager-ip-172-20-52-12.us-east-2.compute.internal started at 2023-01-22 15:55:26 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.700: INFO: 	Container kube-controller-manager ready: true, restart count 2
Jan 22 16:06:33.700: INFO: kube-scheduler-ip-172-20-52-12.us-east-2.compute.internal started at 2023-01-22 15:55:26 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.700: INFO: 	Container kube-scheduler ready: true, restart count 0
Jan 22 16:06:33.700: INFO: cilium-tqjcp started at 2023-01-22 15:56:35 +0000 UTC (1+1 container statuses recorded)
Jan 22 16:06:33.700: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Jan 22 16:06:33.700: INFO: 	Container cilium-agent ready: true, restart count 0
Jan 22 16:06:33.700: INFO: ebs-csi-controller-54b65c69b6-ljh8m started at 2023-01-22 15:56:35 +0000 UTC (0+5 container statuses recorded)
Jan 22 16:06:33.700: INFO: 	Container csi-attacher ready: true, restart count 0
Jan 22 16:06:33.700: INFO: 	Container csi-provisioner ready: true, restart count 0
Jan 22 16:06:33.700: INFO: 	Container csi-resizer ready: true, restart count 0
Jan 22 16:06:33.700: INFO: 	Container ebs-plugin ready: true, restart count 0
Jan 22 16:06:33.700: INFO: 	Container liveness-probe ready: true, restart count 0
Jan 22 16:06:33.700: INFO: dns-controller-7db8b74b99-5rkk4 started at 2023-01-22 15:56:35 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.700: INFO: 	Container dns-controller ready: true, restart count 0
Jan 22 16:06:33.700: INFO: etcd-manager-events-ip-172-20-52-12.us-east-2.compute.internal started at 2023-01-22 15:55:26 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.700: INFO: 	Container etcd-manager ready: true, restart count 0
Jan 22 16:06:33.700: INFO: kube-apiserver-ip-172-20-52-12.us-east-2.compute.internal started at 2023-01-22 15:55:26 +0000 UTC (0+2 container statuses recorded)
Jan 22 16:06:33.700: INFO: 	Container healthcheck ready: true, restart count 0
Jan 22 16:06:33.700: INFO: 	Container kube-apiserver ready: true, restart count 1
Jan 22 16:06:33.700: INFO: kops-controller-c9664 started at 2023-01-22 15:56:35 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.700: INFO: 	Container kops-controller ready: true, restart count 0
Jan 22 16:06:33.700: INFO: ebs-csi-node-288bv started at 2023-01-22 15:56:35 +0000 UTC (0+3 container statuses recorded)
Jan 22 16:06:33.700: INFO: 	Container ebs-plugin ready: true, restart count 0
Jan 22 16:06:33.700: INFO: 	Container liveness-probe ready: true, restart count 0
Jan 22 16:06:33.700: INFO: 	Container node-driver-registrar ready: true, restart count 0
Jan 22 16:06:33.700: INFO: cilium-operator-68bc649ff6-vvr4s started at 2023-01-22 15:56:35 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.700: INFO: 	Container cilium-operator ready: true, restart count 0
Jan 22 16:06:33.700: INFO: etcd-manager-cilium-ip-172-20-52-12.us-east-2.compute.internal started at 2023-01-22 15:55:26 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.700: INFO: 	Container etcd-manager ready: true, restart count 0
Jan 22 16:06:33.700: INFO: etcd-manager-main-ip-172-20-52-12.us-east-2.compute.internal started at 2023-01-22 15:55:26 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:06:33.700: INFO: 	Container etcd-manager ready: true, restart count 0
Jan 22 16:06:33.809: INFO: 
Latency metrics for node ip-172-20-52-12.us-east-2.compute.internal
Jan 22 16:06:33.809: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
�[1mSTEP�[0m: Destroying namespace "kubectl-357" for this suite.

				Click to see stdout/stderrfrom junit_18.xml

Filter through log files

Kubernetes e2e suite [sig-network] Conntrack should be able to preserve UDP traffic when initial unready endpoints get ready 1m30s

go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[sig\-network\]\sConntrack\sshould\sbe\sable\sto\spreserve\sUDP\straffic\swhen\sinitial\sunready\sendpoints\sget\sready$'

/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/conntrack.go:293
Jan 22 16:03:41.594: Failed to connect to backend pod
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/internal/leafnodes/runner.go:113
				[BeforeEach] [sig-network] Conntrack
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185
�[1mSTEP�[0m: Creating a kubernetes client
Jan 22 16:02:12.739: INFO: >>> kubeConfig: /root/.kube/config
�[1mSTEP�[0m: Building a namespace api object, basename conntrack
�[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace
�[1mSTEP�[0m: Waiting for kube-root-ca.crt to be provisioned in namespace
[BeforeEach] [sig-network] Conntrack
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/conntrack.go:96
[It] should be able to preserve UDP traffic when initial unready endpoints get ready
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/conntrack.go:293
�[1mSTEP�[0m: creating a UDP service svc-udp with type=ClusterIP in conntrack-2324
�[1mSTEP�[0m: creating a client pod for probing the service svc-udp
Jan 22 16:02:13.073: INFO: The status of Pod pod-client is Pending, waiting for it to be Running (with Ready = true)
Jan 22 16:02:15.103: INFO: The status of Pod pod-client is Pending, waiting for it to be Running (with Ready = true)
Jan 22 16:02:17.108: INFO: The status of Pod pod-client is Running (Ready = true)
Jan 22 16:02:17.173: INFO: Pod client logs: Sun Jan 22 16:02:15 UTC 2023
Sun Jan 22 16:02:15 UTC 2023 Try: 1

�[1mSTEP�[0m: creating a backend pod pod-server-1 for the service svc-udp
Jan 22 16:02:17.243: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 22 16:02:19.273: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 22 16:02:21.274: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 22 16:02:23.274: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 22 16:02:25.275: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 22 16:02:27.272: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 22 16:02:29.280: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 22 16:02:31.274: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 22 16:02:33.273: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 22 16:02:35.274: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 22 16:02:37.273: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 22 16:02:39.273: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 22 16:02:41.272: INFO: The status of Pod pod-server-1 is Running (Ready = true)
�[1mSTEP�[0m: waiting up to 3m0s for service svc-udp in namespace conntrack-2324 to expose endpoints map[pod-server-1:[80]]
Jan 22 16:02:41.427: INFO: successfully validated that service svc-udp in namespace conntrack-2324 exposes endpoints map[pod-server-1:[80]]
�[1mSTEP�[0m: checking client pod connected to the backend on Node IP 172.20.39.82
Jan 22 16:03:41.594: INFO: Pod client logs: Sun Jan 22 16:02:15 UTC 2023
Sun Jan 22 16:02:15 UTC 2023 Try: 1

Sun Jan 22 16:02:20 UTC 2023 Try: 2

Sun Jan 22 16:02:25 UTC 2023 Try: 3

Sun Jan 22 16:02:30 UTC 2023 Try: 4

Sun Jan 22 16:02:35 UTC 2023 Try: 5

Sun Jan 22 16:02:40 UTC 2023 Try: 6

Sun Jan 22 16:02:45 UTC 2023 Try: 7

Sun Jan 22 16:02:50 UTC 2023 Try: 8

Sun Jan 22 16:02:55 UTC 2023 Try: 9

Sun Jan 22 16:03:00 UTC 2023 Try: 10

Sun Jan 22 16:03:05 UTC 2023 Try: 11

Sun Jan 22 16:03:10 UTC 2023 Try: 12

Sun Jan 22 16:03:15 UTC 2023 Try: 13

Sun Jan 22 16:03:20 UTC 2023 Try: 14

Sun Jan 22 16:03:25 UTC 2023 Try: 15

Sun Jan 22 16:03:30 UTC 2023 Try: 16

Sun Jan 22 16:03:35 UTC 2023 Try: 17

Sun Jan 22 16:03:40 UTC 2023 Try: 18

Jan 22 16:03:41.594: FAIL: Failed to connect to backend pod

Full Stack Trace
k8s.io/kubernetes/test/e2e.RunE2ETests(0x24eec17?)
	_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:133 +0x6b7
k8s.io/kubernetes/test/e2e.TestE2E(0x24602d9?)
	_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:136 +0x19
testing.tRunner(0xc0008a3040, 0x72ecb90)
	/usr/local/go/src/testing/testing.go:1446 +0x10b
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:1493 +0x35f
[AfterEach] [sig-network] Conntrack
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186
�[1mSTEP�[0m: Collecting events from namespace "conntrack-2324".
�[1mSTEP�[0m: Found 11 events.
Jan 22 16:03:41.625: INFO: At 2023-01-22 16:02:13 +0000 UTC - event for pod-client: {default-scheduler } Scheduled: Successfully assigned conntrack-2324/pod-client to ip-172-20-39-104.us-east-2.compute.internal
Jan 22 16:03:41.625: INFO: At 2023-01-22 16:02:15 +0000 UTC - event for pod-client: {kubelet ip-172-20-39-104.us-east-2.compute.internal} Pulled: Container image "k8s.gcr.io/e2e-test-images/agnhost:2.39" already present on machine
Jan 22 16:03:41.625: INFO: At 2023-01-22 16:02:15 +0000 UTC - event for pod-client: {kubelet ip-172-20-39-104.us-east-2.compute.internal} Created: Created container pod-client
Jan 22 16:03:41.625: INFO: At 2023-01-22 16:02:15 +0000 UTC - event for pod-client: {kubelet ip-172-20-39-104.us-east-2.compute.internal} Started: Started container pod-client
Jan 22 16:03:41.625: INFO: At 2023-01-22 16:02:17 +0000 UTC - event for pod-server-1: {default-scheduler } Scheduled: Successfully assigned conntrack-2324/pod-server-1 to ip-172-20-39-82.us-east-2.compute.internal
Jan 22 16:03:41.625: INFO: At 2023-01-22 16:02:18 +0000 UTC - event for pod-server-1: {kubelet ip-172-20-39-82.us-east-2.compute.internal} Pulled: Container image "k8s.gcr.io/e2e-test-images/busybox:1.29-2" already present on machine
Jan 22 16:03:41.625: INFO: At 2023-01-22 16:02:18 +0000 UTC - event for pod-server-1: {kubelet ip-172-20-39-82.us-east-2.compute.internal} Created: Created container init
Jan 22 16:03:41.625: INFO: At 2023-01-22 16:02:18 +0000 UTC - event for pod-server-1: {kubelet ip-172-20-39-82.us-east-2.compute.internal} Started: Started container init
Jan 22 16:03:41.625: INFO: At 2023-01-22 16:02:35 +0000 UTC - event for pod-server-1: {kubelet ip-172-20-39-82.us-east-2.compute.internal} Pulled: Container image "k8s.gcr.io/e2e-test-images/agnhost:2.39" already present on machine
Jan 22 16:03:41.625: INFO: At 2023-01-22 16:02:35 +0000 UTC - event for pod-server-1: {kubelet ip-172-20-39-82.us-east-2.compute.internal} Created: Created container agnhost-container
Jan 22 16:03:41.625: INFO: At 2023-01-22 16:02:35 +0000 UTC - event for pod-server-1: {kubelet ip-172-20-39-82.us-east-2.compute.internal} Started: Started container agnhost-container
Jan 22 16:03:41.654: INFO: POD           NODE                                         PHASE    GRACE  CONDITIONS
Jan 22 16:03:41.654: INFO: pod-client    ip-172-20-39-104.us-east-2.compute.internal  Running         [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-22 16:02:13 +0000 UTC  } {Ready True 0001-01-01 00:00:00 +0000 UTC 2023-01-22 16:02:15 +0000 UTC  } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-01-22 16:02:15 +0000 UTC  } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-22 16:02:13 +0000 UTC  }]
Jan 22 16:03:41.654: INFO: pod-server-1  ip-172-20-39-82.us-east-2.compute.internal   Running         [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-22 16:02:35 +0000 UTC  } {Ready True 0001-01-01 00:00:00 +0000 UTC 2023-01-22 16:02:36 +0000 UTC  } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-01-22 16:02:36 +0000 UTC  } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-22 16:02:17 +0000 UTC  }]
Jan 22 16:03:41.654: INFO: 
Jan 22 16:03:41.757: INFO: 
Logging node info for node ip-172-20-39-104.us-east-2.compute.internal
Jan 22 16:03:41.786: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-39-104.us-east-2.compute.internal    94a127fa-2d03-4fae-878f-95cc778a7072 6485 0 2023-01-22 15:57:23 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-east-2 failure-domain.beta.kubernetes.io/zone:us-east-2a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-39-104.us-east-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-east-2a topology.hostpath.csi/node:ip-172-20-39-104.us-east-2.compute.internal topology.kubernetes.io/region:us-east-2 topology.kubernetes.io/zone:us-east-2a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-446":"ip-172-20-39-104.us-east-2.compute.internal","ebs.csi.aws.com":"i-058f72b2ec21c63a8"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kops-controller Update v1 2023-01-22 15:57:23 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-22 15:57:23 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {kubelet Update v1 2023-01-22 15:57:23 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2023-01-22 16:02:09 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-22 16:02:16 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status} {e2e.test Update v1 2023-01-22 16:02:30 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} }]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///us-east-2a/i-058f72b2ec21c63a8,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{47441653760 0} {<nil>} 46329740Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4054806528 0} {<nil>} 3959772Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{42697488314 0} {<nil>} 42697488314 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3949948928 0} {<nil>} 3857372Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-22 16:03:03 +0000 UTC,LastTransitionTime:2023-01-22 15:57:23 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-22 16:03:03 +0000 UTC,LastTransitionTime:2023-01-22 15:57:23 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-22 16:03:03 +0000 UTC,LastTransitionTime:2023-01-22 15:57:23 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-22 16:03:03 +0000 UTC,LastTransitionTime:2023-01-22 15:57:44 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.39.104,},NodeAddress{Type:ExternalIP,Address:3.133.79.156,},NodeAddress{Type:Hostname,Address:ip-172-20-39-104.us-east-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-39-104.us-east-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-133-79-156.us-east-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2455e84fc0e92553e5c201c3d73752,SystemUUID:ec2455e8-4fc0-e925-53e5-c201c3d73752,BootID:baa49577-2567-45b4-b63f-1f65be12643c,KernelVersion:5.15.86-flatcar,OSImage:Flatcar Container Linux by Kinvolk 3446.1.0 (Oklo),ContainerRuntimeVersion:docker://20.10.21,KubeletVersion:v1.23.16,KubeProxyVersion:v1.23.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456551323,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126853136,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:99c0d6f1ad24a1aa1905d9c6534d193f268f7b23f9add2ae6bb41f31094bdd5c k8s.gcr.io/e2e-test-images/nautilus:1.5],SizeBytes:124738681,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123774598,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.16 registry.k8s.io/kube-proxy-amd64:v1.23.16],SizeBytes:110829687,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:94469968,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56363051,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53940090,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53515772,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53483352,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:50249337,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:47903915,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29972082,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19575418,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18683218,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17605286,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17159270,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16028126,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-446^2962d889-9a6e-11ed-aca1-b240603228b3 kubernetes.io/csi/ebs.csi.aws.com^vol-00f9ca3f8a10b690b],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-00f9ca3f8a10b690b,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-446^2962d889-9a6e-11ed-aca1-b240603228b3,DevicePath:,},},Config:nil,},}
Jan 22 16:03:41.787: INFO: 
Logging kubelet events for node ip-172-20-39-104.us-east-2.compute.internal
Jan 22 16:03:41.828: INFO: 
Logging pods the kubelet thinks is on node ip-172-20-39-104.us-east-2.compute.internal
Jan 22 16:03:41.908: INFO: cilium-cl25d started at 2023-01-22 15:57:24 +0000 UTC (1+1 container statuses recorded)
Jan 22 16:03:41.908: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Jan 22 16:03:41.908: INFO: 	Container cilium-agent ready: false, restart count 1
Jan 22 16:03:41.908: INFO: pod-d0184f4b-c93e-456b-bc7d-3eaf544b4775 started at 2023-01-22 16:03:21 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:41.908: INFO: 	Container write-pod ready: false, restart count 0
Jan 22 16:03:41.908: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-265ss started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:41.908: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: false, restart count 0
Jan 22 16:03:41.908: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-n866x started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:41.908: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: false, restart count 0
Jan 22 16:03:41.908: INFO: csi-mockplugin-attacher-0 started at 2023-01-22 16:02:37 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:41.908: INFO: 	Container csi-attacher ready: false, restart count 0
Jan 22 16:03:41.908: INFO: ebs-csi-node-8fr2s started at 2023-01-22 15:57:24 +0000 UTC (0+3 container statuses recorded)
Jan 22 16:03:41.908: INFO: 	Container ebs-plugin ready: true, restart count 0
Jan 22 16:03:41.908: INFO: 	Container liveness-probe ready: true, restart count 0
Jan 22 16:03:41.908: INFO: 	Container node-driver-registrar ready: true, restart count 0
Jan 22 16:03:41.908: INFO: csi-mockplugin-attacher-0 started at 2023-01-22 16:03:27 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:41.908: INFO: 	Container csi-attacher ready: false, restart count 0
Jan 22 16:03:41.908: INFO: csi-mockplugin-resizer-0 started at 2023-01-22 16:02:37 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:41.908: INFO: 	Container csi-resizer ready: false, restart count 0
Jan 22 16:03:41.908: INFO: test-rolling-update-with-lb-79bf9f6988-6rvkk started at 2023-01-22 16:01:39 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:41.908: INFO: 	Container agnhost ready: true, restart count 0
Jan 22 16:03:41.908: INFO: pod-client started at 2023-01-22 16:02:13 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:41.908: INFO: 	Container pod-client ready: true, restart count 0
Jan 22 16:03:41.908: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-b44bm started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:41.908: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: false, restart count 0
Jan 22 16:03:41.908: INFO: coredns-867df8f45c-7d727 started at 2023-01-22 15:57:44 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:41.908: INFO: 	Container coredns ready: true, restart count 0
Jan 22 16:03:41.908: INFO: replace-27906722-cz55r started at 2023-01-22 16:02:00 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:41.908: INFO: 	Container c ready: false, restart count 0
Jan 22 16:03:41.908: INFO: agnhost-primary-69cb998d54-brrxv started at 2023-01-22 16:03:14 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:41.908: INFO: 	Container primary ready: false, restart count 0
Jan 22 16:03:41.908: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-zltrj started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:41.908: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: false, restart count 0
Jan 22 16:03:41.908: INFO: pod-ephm-test-projected-bkvb started at 2023-01-22 16:02:03 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:41.908: INFO: 	Container test-container-subpath-projected-bkvb ready: false, restart count 0
Jan 22 16:03:41.908: INFO: pod-subpath-test-inlinevolume-r9xz started at 2023-01-22 16:02:21 +0000 UTC (2+1 container statuses recorded)
Jan 22 16:03:41.908: INFO: 	Init container init-volume-inlinevolume-r9xz ready: false, restart count 0
Jan 22 16:03:41.908: INFO: 	Init container test-init-volume-inlinevolume-r9xz ready: false, restart count 0
Jan 22 16:03:41.908: INFO: 	Container test-container-subpath-inlinevolume-r9xz ready: false, restart count 0
Jan 22 16:03:41.908: INFO: pause-pod-1 started at 2023-01-22 16:01:54 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:41.908: INFO: 	Container agnhost-container ready: true, restart count 0
Jan 22 16:03:41.908: INFO: inline-volume-tester-jpszl started at 2023-01-22 16:02:25 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:41.908: INFO: 	Container csi-volume-tester ready: false, restart count 0
Jan 22 16:03:41.908: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-zkthv started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:41.908: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: false, restart count 0
Jan 22 16:03:41.908: INFO: ss-1 started at 2023-01-22 16:02:19 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:41.908: INFO: 	Container webserver ready: false, restart count 0
Jan 22 16:03:41.908: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-ltqph started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:41.908: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: false, restart count 0
Jan 22 16:03:41.908: INFO: csi-mockplugin-0 started at 2023-01-22 16:02:37 +0000 UTC (0+3 container statuses recorded)
Jan 22 16:03:41.908: INFO: 	Container csi-provisioner ready: false, restart count 0
Jan 22 16:03:41.908: INFO: 	Container driver-registrar ready: false, restart count 0
Jan 22 16:03:41.908: INFO: 	Container mock ready: false, restart count 0
Jan 22 16:03:41.908: INFO: hostexec-ip-172-20-39-104.us-east-2.compute.internal-5d8rz started at 2023-01-22 16:03:10 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:41.908: INFO: 	Container agnhost-container ready: true, restart count 0
Jan 22 16:03:41.908: INFO: netserver-0 started at 2023-01-22 16:03:41 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:41.908: INFO: 	Container webserver ready: false, restart count 0
Jan 22 16:03:41.908: INFO: coredns-autoscaler-557ccb4c66-44k6v started at 2023-01-22 15:57:44 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:41.908: INFO: 	Container autoscaler ready: true, restart count 0
Jan 22 16:03:41.908: INFO: csi-hostpathplugin-0 started at 2023-01-22 16:02:09 +0000 UTC (0+7 container statuses recorded)
Jan 22 16:03:41.908: INFO: 	Container csi-attacher ready: true, restart count 0
Jan 22 16:03:41.908: INFO: 	Container csi-provisioner ready: true, restart count 0
Jan 22 16:03:41.908: INFO: 	Container csi-resizer ready: true, restart count 0
Jan 22 16:03:41.908: INFO: 	Container csi-snapshotter ready: true, restart count 0
Jan 22 16:03:41.908: INFO: 	Container hostpath ready: true, restart count 0
Jan 22 16:03:41.908: INFO: 	Container liveness-probe ready: true, restart count 0
Jan 22 16:03:41.908: INFO: 	Container node-driver-registrar ready: true, restart count 0
Jan 22 16:03:41.908: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-mn2wn started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:41.908: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: false, restart count 0
Jan 22 16:03:41.908: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-dmxrv started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:41.908: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: false, restart count 0
Jan 22 16:03:41.908: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-7sknq started at 2023-01-22 16:02:31 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:41.908: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: false, restart count 0
Jan 22 16:03:41.908: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-hrlf4 started at 2023-01-22 16:02:31 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:41.908: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: false, restart count 0
Jan 22 16:03:41.908: INFO: netserver-0 started at 2023-01-22 16:02:31 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:41.908: INFO: 	Container webserver ready: false, restart count 0
Jan 22 16:03:41.908: INFO: csi-mockplugin-0 started at 2023-01-22 16:03:27 +0000 UTC (0+3 container statuses recorded)
Jan 22 16:03:41.908: INFO: 	Container csi-provisioner ready: false, restart count 0
Jan 22 16:03:41.908: INFO: 	Container driver-registrar ready: false, restart count 0
Jan 22 16:03:41.908: INFO: 	Container mock ready: false, restart count 0
Jan 22 16:03:42.117: INFO: 
Latency metrics for node ip-172-20-39-104.us-east-2.compute.internal
Jan 22 16:03:42.118: INFO: 
Logging node info for node ip-172-20-39-82.us-east-2.compute.internal
Jan 22 16:03:42.148: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-39-82.us-east-2.compute.internal    fab1195f-32aa-4044-ac45-046bbfde063a 6654 0 2023-01-22 15:57:34 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-east-2 failure-domain.beta.kubernetes.io/zone:us-east-2a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-39-82.us-east-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-east-2a topology.kubernetes.io/region:us-east-2 topology.kubernetes.io/zone:us-east-2a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-519":"csi-mock-csi-mock-volumes-519","ebs.csi.aws.com":"i-0a00ce72536f23489"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kops-controller Update v1 2023-01-22 15:57:34 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-22 15:57:34 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2023-01-22 15:57:34 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kubelet Update v1 2023-01-22 16:01:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2023-01-22 16:01:40 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {e2e.test Update v1 2023-01-22 16:02:30 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} }]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///us-east-2a/i-0a00ce72536f23489,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{47441653760 0} {<nil>} 46329740Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4054806528 0} {<nil>} 3959772Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{42697488314 0} {<nil>} 42697488314 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3949948928 0} {<nil>} 3857372Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-22 16:03:11 +0000 UTC,LastTransitionTime:2023-01-22 15:57:34 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-22 16:03:11 +0000 UTC,LastTransitionTime:2023-01-22 15:57:34 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-22 16:03:11 +0000 UTC,LastTransitionTime:2023-01-22 15:57:34 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-22 16:03:11 +0000 UTC,LastTransitionTime:2023-01-22 15:57:54 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.39.82,},NodeAddress{Type:ExternalIP,Address:3.14.247.52,},NodeAddress{Type:Hostname,Address:ip-172-20-39-82.us-east-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-39-82.us-east-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-14-247-52.us-east-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bb4ac3588554994a702eabb6be703,SystemUUID:ec2bb4ac-3588-5549-94a7-02eabb6be703,BootID:82a481e4-e615-4371-bbc2-0b7f74e99966,KernelVersion:5.15.86-flatcar,OSImage:Flatcar Container Linux by Kinvolk 3446.1.0 (Oklo),ContainerRuntimeVersion:docker://20.10.21,KubeletVersion:v1.23.16,KubeProxyVersion:v1.23.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456551323,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:253346057,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126853136,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:99c0d6f1ad24a1aa1905d9c6534d193f268f7b23f9add2ae6bb41f31094bdd5c k8s.gcr.io/e2e-test-images/nautilus:1.5],SizeBytes:124738681,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123774598,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.16 registry.k8s.io/kube-proxy-amd64:v1.23.16],SizeBytes:110829687,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:94469968,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56366111,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53484608,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:50249337,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19646681,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19575418,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17664775,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17605286,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16028126,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-519^4 kubernetes.io/csi/ebs.csi.aws.com^vol-07558a7d8ec7e2f44],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-07558a7d8ec7e2f44,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-519^4,DevicePath:,},},Config:nil,},}
Jan 22 16:03:42.149: INFO: 
Logging kubelet events for node ip-172-20-39-82.us-east-2.compute.internal
Jan 22 16:03:42.202: INFO: 
Logging pods the kubelet thinks is on node ip-172-20-39-82.us-east-2.compute.internal
Jan 22 16:03:42.243: INFO: pod-subpath-test-preprovisionedpv-ctkz started at 2023-01-22 16:03:36 +0000 UTC (2+1 container statuses recorded)
Jan 22 16:03:42.243: INFO: 	Init container init-volume-preprovisionedpv-ctkz ready: true, restart count 0
Jan 22 16:03:42.243: INFO: 	Init container test-init-volume-preprovisionedpv-ctkz ready: true, restart count 0
Jan 22 16:03:42.243: INFO: 	Container test-container-subpath-preprovisionedpv-ctkz ready: false, restart count 0
Jan 22 16:03:42.243: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-tbzwq started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:42.243: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:03:42.243: INFO: csi-mockplugin-0 started at 2023-01-22 16:01:59 +0000 UTC (0+3 container statuses recorded)
Jan 22 16:03:42.243: INFO: 	Container csi-provisioner ready: true, restart count 0
Jan 22 16:03:42.243: INFO: 	Container driver-registrar ready: true, restart count 0
Jan 22 16:03:42.243: INFO: 	Container mock ready: true, restart count 0
Jan 22 16:03:42.243: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-f548n started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:42.243: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:03:42.243: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-4pwz8 started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:42.243: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:03:42.243: INFO: cilium-dpwnn started at 2023-01-22 15:57:34 +0000 UTC (1+1 container statuses recorded)
Jan 22 16:03:42.243: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Jan 22 16:03:42.243: INFO: 	Container cilium-agent ready: true, restart count 1
Jan 22 16:03:42.243: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-hzsmm started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:42.243: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:03:42.243: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-dwhkt started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:42.243: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:03:42.243: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-m8pkb started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:42.243: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:03:42.243: INFO: netserver-1 started at 2023-01-22 16:03:41 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:42.243: INFO: 	Container webserver ready: false, restart count 0
Jan 22 16:03:42.243: INFO: csi-mockplugin-0 started at 2023-01-22 16:03:37 +0000 UTC (0+4 container statuses recorded)
Jan 22 16:03:42.243: INFO: 	Container busybox ready: true, restart count 0
Jan 22 16:03:42.243: INFO: 	Container csi-provisioner ready: true, restart count 0
Jan 22 16:03:42.243: INFO: 	Container driver-registrar ready: true, restart count 0
Jan 22 16:03:42.243: INFO: 	Container mock ready: true, restart count 0
Jan 22 16:03:42.243: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-pr6wn started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:42.243: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:03:42.243: INFO: ebs-csi-node-5xvpn started at 2023-01-22 15:57:34 +0000 UTC (0+3 container statuses recorded)
Jan 22 16:03:42.243: INFO: 	Container ebs-plugin ready: true, restart count 0
Jan 22 16:03:42.243: INFO: 	Container liveness-probe ready: true, restart count 0
Jan 22 16:03:42.243: INFO: 	Container node-driver-registrar ready: true, restart count 0
Jan 22 16:03:42.243: INFO: pod-configmaps-74dae880-761e-4c06-93b9-70056d19519c started at 2023-01-22 16:02:19 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:42.243: INFO: 	Container agnhost-container ready: true, restart count 0
Jan 22 16:03:42.243: INFO: frontend-5c4f744f96-7tjxh started at 2023-01-22 16:03:14 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:42.243: INFO: 	Container guestbook-frontend ready: true, restart count 0
Jan 22 16:03:42.243: INFO: netserver-1 started at 2023-01-22 16:02:31 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:42.243: INFO: 	Container webserver ready: true, restart count 0
Jan 22 16:03:42.243: INFO: coredns-867df8f45c-7m6hf started at 2023-01-22 15:58:04 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:42.243: INFO: 	Container coredns ready: true, restart count 0
Jan 22 16:03:42.243: INFO: ss-0 started at 2023-01-22 16:01:47 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:42.243: INFO: 	Container webserver ready: true, restart count 0
Jan 22 16:03:42.243: INFO: pod-server-1 started at 2023-01-22 16:02:17 +0000 UTC (1+1 container statuses recorded)
Jan 22 16:03:42.243: INFO: 	Init container init ready: true, restart count 0
Jan 22 16:03:42.243: INFO: 	Container agnhost-container ready: true, restart count 0
Jan 22 16:03:42.243: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-s2z7v started at 2023-01-22 16:02:31 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:42.243: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:03:42.243: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-k72n5 started at 2023-01-22 16:02:31 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:42.243: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:03:42.243: INFO: csi-mockplugin-attacher-0 started at 2023-01-22 16:01:59 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:42.243: INFO: 	Container csi-attacher ready: true, restart count 0
Jan 22 16:03:42.243: INFO: test-rolling-update-with-lb-79bf9f6988-kwp9d started at 2023-01-22 16:01:39 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:42.243: INFO: 	Container agnhost ready: true, restart count 0
Jan 22 16:03:42.243: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-cvg4m started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:42.244: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:03:42.244: INFO: hostexec-ip-172-20-39-82.us-east-2.compute.internal-5bc2k started at 2023-01-22 16:03:21 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:42.244: INFO: 	Container agnhost-container ready: true, restart count 0
Jan 22 16:03:42.656: INFO: 
Latency metrics for node ip-172-20-39-82.us-east-2.compute.internal
Jan 22 16:03:42.656: INFO: 
Logging node info for node ip-172-20-40-246.us-east-2.compute.internal
Jan 22 16:03:42.686: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-40-246.us-east-2.compute.internal    a4283492-f7a9-4ac6-aaab-f667bea07522 6748 0 2023-01-22 15:57:26 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-east-2 failure-domain.beta.kubernetes.io/zone:us-east-2a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-40-246.us-east-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-east-2a topology.hostpath.csi/node:ip-172-20-40-246.us-east-2.compute.internal topology.kubernetes.io/region:us-east-2 topology.kubernetes.io/zone:us-east-2a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-6352":"ip-172-20-40-246.us-east-2.compute.internal","csi-mock-csi-mock-volumes-9016":"csi-mock-csi-mock-volumes-9016","ebs.csi.aws.com":"i-0a5034e547859f6df"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kubelet Update v1 2023-01-22 15:57:26 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kops-controller Update v1 2023-01-22 15:57:27 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-22 15:57:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kube-controller-manager Update v1 2023-01-22 16:01:06 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-22 16:01:20 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status} {e2e.test Update v1 2023-01-22 16:02:30 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} }]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///us-east-2a/i-0a5034e547859f6df,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{47441653760 0} {<nil>} 46329740Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4054806528 0} {<nil>} 3959772Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{42697488314 0} {<nil>} 42697488314 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3949948928 0} {<nil>} 3857372Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-22 16:03:13 +0000 UTC,LastTransitionTime:2023-01-22 15:57:26 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-22 16:03:13 +0000 UTC,LastTransitionTime:2023-01-22 15:57:26 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-22 16:03:13 +0000 UTC,LastTransitionTime:2023-01-22 15:57:26 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-22 16:03:13 +0000 UTC,LastTransitionTime:2023-01-22 15:57:47 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.40.246,},NodeAddress{Type:ExternalIP,Address:3.22.242.15,},NodeAddress{Type:Hostname,Address:ip-172-20-40-246.us-east-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-40-246.us-east-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-22-242-15.us-east-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2ac3af6059e59b70f877f01430277f,SystemUUID:ec2ac3af-6059-e59b-70f8-77f01430277f,BootID:f03e4ca2-c137-40c8-ba1b-f79f8d612996,KernelVersion:5.15.86-flatcar,OSImage:Flatcar Container Linux by Kinvolk 3446.1.0 (Oklo),ContainerRuntimeVersion:docker://20.10.21,KubeletVersion:v1.23.16,KubeProxyVersion:v1.23.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456551323,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126853136,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123774598,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.16 registry.k8s.io/kube-proxy-amd64:v1.23.16],SizeBytes:110829687,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:94469968,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56366111,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56363051,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53940090,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53515772,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53483352,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:b9e2958a3dd879e3cf11142228c6d073d0fc4ea2e857c3be6f4fb0ab5fb2c937 k8s.gcr.io/e2e-test-images/nonroot:1.2],SizeBytes:42320060,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29972082,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19646681,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19575418,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18683218,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17664775,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17605286,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17159270,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16028126,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-6352^07fde2bd-9a6e-11ed-979a-5ec47c07e900],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-6352^07fde2bd-9a6e-11ed-979a-5ec47c07e900,DevicePath:,},},Config:nil,},}
Jan 22 16:03:42.686: INFO: 
Logging kubelet events for node ip-172-20-40-246.us-east-2.compute.internal
Jan 22 16:03:42.719: INFO: 
Logging pods the kubelet thinks is on node ip-172-20-40-246.us-east-2.compute.internal
Jan 22 16:03:42.757: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-26ll5 started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:42.758: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:03:42.758: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-2bwv6 started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:42.758: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:03:42.758: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-jp7lk started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:42.758: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:03:42.758: INFO: frontend-5c4f744f96-zz7vx started at 2023-01-22 16:03:14 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:42.758: INFO: 	Container guestbook-frontend ready: true, restart count 0
Jan 22 16:03:42.758: INFO: webserver-pod started at 2023-01-22 16:01:22 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:42.758: INFO: 	Container agnhost ready: false, restart count 0
Jan 22 16:03:42.758: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-pdmh6 started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:42.758: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:03:42.758: INFO: csi-hostpathplugin-0 started at 2023-01-22 16:01:10 +0000 UTC (0+7 container statuses recorded)
Jan 22 16:03:42.758: INFO: 	Container csi-attacher ready: true, restart count 0
Jan 22 16:03:42.758: INFO: 	Container csi-provisioner ready: true, restart count 0
Jan 22 16:03:42.758: INFO: 	Container csi-resizer ready: true, restart count 0
Jan 22 16:03:42.758: INFO: 	Container csi-snapshotter ready: true, restart count 0
Jan 22 16:03:42.758: INFO: 	Container hostpath ready: true, restart count 0
Jan 22 16:03:42.758: INFO: 	Container liveness-probe ready: true, restart count 0
Jan 22 16:03:42.758: INFO: 	Container node-driver-registrar ready: true, restart count 0
Jan 22 16:03:42.758: INFO: netserver-2 started at 2023-01-22 16:03:41 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:42.758: INFO: 	Container webserver ready: false, restart count 0
Jan 22 16:03:42.758: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-2xdb2 started at 2023-01-22 16:02:31 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:42.758: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:03:42.758: INFO: cilium-s627s started at 2023-01-22 15:57:27 +0000 UTC (1+1 container statuses recorded)
Jan 22 16:03:42.758: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Jan 22 16:03:42.758: INFO: 	Container cilium-agent ready: true, restart count 2
Jan 22 16:03:42.758: INFO: csi-mockplugin-0 started at 2023-01-22 16:02:20 +0000 UTC (0+3 container statuses recorded)
Jan 22 16:03:42.758: INFO: 	Container csi-provisioner ready: true, restart count 0
Jan 22 16:03:42.758: INFO: 	Container driver-registrar ready: true, restart count 0
Jan 22 16:03:42.758: INFO: 	Container mock ready: true, restart count 0
Jan 22 16:03:42.758: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-lrc95 started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:42.758: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:03:42.758: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-bgclt started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:42.758: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:03:42.758: INFO: test-rolling-update-with-lb-79bf9f6988-lvxhx started at 2023-01-22 16:01:39 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:42.758: INFO: 	Container agnhost ready: true, restart count 0
Jan 22 16:03:42.758: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-2kqq5 started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:42.758: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:03:42.758: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-jp49t started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:42.758: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:03:42.758: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-tg6vg started at 2023-01-22 16:02:31 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:42.758: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:03:42.758: INFO: hostexec-ip-172-20-40-246.us-east-2.compute.internal-q8jzm started at 2023-01-22 16:03:06 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:42.758: INFO: 	Container agnhost-container ready: true, restart count 0
Jan 22 16:03:42.758: INFO: pod-client started at 2023-01-22 16:03:40 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:42.758: INFO: 	Container pod-client ready: true, restart count 0
Jan 22 16:03:42.758: INFO: ebs-csi-node-vrjbd started at 2023-01-22 15:57:27 +0000 UTC (0+3 container statuses recorded)
Jan 22 16:03:42.758: INFO: 	Container ebs-plugin ready: true, restart count 0
Jan 22 16:03:42.758: INFO: 	Container liveness-probe ready: true, restart count 0
Jan 22 16:03:42.758: INFO: 	Container node-driver-registrar ready: true, restart count 0
Jan 22 16:03:42.758: INFO: agnhost-replica-6f5fb76474-pmnld started at 2023-01-22 16:03:14 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:42.758: INFO: 	Container replica ready: true, restart count 0
Jan 22 16:03:42.758: INFO: netserver-2 started at 2023-01-22 16:02:31 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:42.758: INFO: 	Container webserver ready: true, restart count 0
Jan 22 16:03:42.758: INFO: pvc-volume-tester-2rfj9 started at 2023-01-22 16:02:32 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:42.758: INFO: 	Container volume-tester ready: true, restart count 0
Jan 22 16:03:42.758: INFO: pod-subpath-test-preprovisionedpv-7qlf started at 2023-01-22 16:03:22 +0000 UTC (1+1 container statuses recorded)
Jan 22 16:03:42.758: INFO: 	Init container init-volume-preprovisionedpv-7qlf ready: true, restart count 0
Jan 22 16:03:42.758: INFO: 	Container test-container-subpath-preprovisionedpv-7qlf ready: true, restart count 0
Jan 22 16:03:42.903: INFO: 
Latency metrics for node ip-172-20-40-246.us-east-2.compute.internal
Jan 22 16:03:42.903: INFO: 
Logging node info for node ip-172-20-45-186.us-east-2.compute.internal
Jan 22 16:03:42.936: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-45-186.us-east-2.compute.internal    d0ea7688-ae6f-4b0b-b4c0-87950ac02237 7226 0 2023-01-22 15:57:24 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-east-2 failure-domain.beta.kubernetes.io/zone:us-east-2a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-45-186.us-east-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-east-2a topology.hostpath.csi/node:ip-172-20-45-186.us-east-2.compute.internal topology.kubernetes.io/region:us-east-2 topology.kubernetes.io/zone:us-east-2a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-1466":"csi-mock-csi-mock-volumes-1466","ebs.csi.aws.com":"i-0ce264dea7066c689"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kops-controller Update v1 2023-01-22 15:57:24 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-01-22 15:57:24 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2023-01-22 15:57:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2023-01-22 16:02:00 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status} {e2e.test Update v1 2023-01-22 16:02:30 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} }]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///us-east-2a/i-0ce264dea7066c689,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{47441653760 0} {<nil>} 46329740Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4054806528 0} {<nil>} 3959772Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{42697488314 0} {<nil>} 42697488314 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3949948928 0} {<nil>} 3857372Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-22 16:03:32 +0000 UTC,LastTransitionTime:2023-01-22 15:57:24 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-22 16:03:32 +0000 UTC,LastTransitionTime:2023-01-22 15:57:24 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-22 16:03:32 +0000 UTC,LastTransitionTime:2023-01-22 15:57:24 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-22 16:03:32 +0000 UTC,LastTransitionTime:2023-01-22 15:57:55 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.45.186,},NodeAddress{Type:ExternalIP,Address:18.118.166.212,},NodeAddress{Type:Hostname,Address:ip-172-20-45-186.us-east-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-45-186.us-east-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-118-166-212.us-east-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2308cb46db6b634f318d31f03adcbc,SystemUUID:ec2308cb-46db-6b63-4f31-8d31f03adcbc,BootID:c1380477-0aee-480e-8dd1-20b7d9813f9d,KernelVersion:5.15.86-flatcar,OSImage:Flatcar Container Linux by Kinvolk 3446.1.0 (Oklo),ContainerRuntimeVersion:docker://20.10.21,KubeletVersion:v1.23.16,KubeProxyVersion:v1.23.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456551323,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126853136,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123774598,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.16 registry.k8s.io/kube-proxy-amd64:v1.23.16],SizeBytes:110829687,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:94469968,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56366111,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56363051,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53940090,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53515772,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53484608,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53483352,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:49160266,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29972082,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19646681,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19575418,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18683218,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17664775,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17605286,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17159270,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16028126,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:4202236,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-1466^4],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Jan 22 16:03:42.937: INFO: 
Logging kubelet events for node ip-172-20-45-186.us-east-2.compute.internal
Jan 22 16:03:42.970: INFO: 
Logging pods the kubelet thinks is on node ip-172-20-45-186.us-east-2.compute.internal
Jan 22 16:03:43.005: INFO: csi-mockplugin-0 started at 2023-01-22 16:00:55 +0000 UTC (0+3 container statuses recorded)
Jan 22 16:03:43.006: INFO: 	Container csi-provisioner ready: true, restart count 0
Jan 22 16:03:43.006: INFO: 	Container driver-registrar ready: true, restart count 0
Jan 22 16:03:43.006: INFO: 	Container mock ready: true, restart count 0
Jan 22 16:03:43.006: INFO: ebs-csi-node-45fpb started at 2023-01-22 15:57:25 +0000 UTC (0+3 container statuses recorded)
Jan 22 16:03:43.006: INFO: 	Container ebs-plugin ready: true, restart count 0
Jan 22 16:03:43.006: INFO: 	Container liveness-probe ready: true, restart count 0
Jan 22 16:03:43.006: INFO: 	Container node-driver-registrar ready: true, restart count 0
Jan 22 16:03:43.006: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-gmj8t started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:43.006: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:03:43.006: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-qjt7c started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:43.006: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:03:43.006: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-fq5q8 started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:43.006: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:03:43.006: INFO: csi-mockplugin-attacher-0 started at 2023-01-22 16:00:55 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:43.006: INFO: 	Container csi-attacher ready: true, restart count 0
Jan 22 16:03:43.006: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-7xqsc started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:43.006: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:03:43.006: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-7f88x started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:43.006: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:03:43.006: INFO: cilium-zt6nc started at 2023-01-22 15:57:25 +0000 UTC (1+1 container statuses recorded)
Jan 22 16:03:43.006: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Jan 22 16:03:43.006: INFO: 	Container cilium-agent ready: true, restart count 2
Jan 22 16:03:43.006: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-8d2c7 started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:43.006: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:03:43.006: INFO: netserver-3 started at 2023-01-22 16:02:31 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:43.006: INFO: 	Container webserver ready: true, restart count 0
Jan 22 16:03:43.006: INFO: netserver-3 started at 2023-01-22 16:03:41 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:43.006: INFO: 	Container webserver ready: false, restart count 0
Jan 22 16:03:43.006: INFO: agnhost-replica-6f5fb76474-v7nq2 started at 2023-01-22 16:03:15 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:43.006: INFO: 	Container replica ready: true, restart count 1
Jan 22 16:03:43.006: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-kxcmp started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:43.006: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:03:43.006: INFO: pvc-volume-tester-srzvk started at 2023-01-22 16:01:56 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:43.006: INFO: 	Container volume-tester ready: false, restart count 0
Jan 22 16:03:43.006: INFO: test-pod started at 2023-01-22 16:02:12 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:43.006: INFO: 	Container test-container ready: false, restart count 0
Jan 22 16:03:43.006: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-26q8s started at 2023-01-22 16:02:31 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:43.006: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:03:43.006: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-8q6b2 started at 2023-01-22 16:02:31 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:43.006: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:03:43.006: INFO: frontend-5c4f744f96-lbzp6 started at 2023-01-22 16:03:14 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:43.006: INFO: 	Container guestbook-frontend ready: true, restart count 0
Jan 22 16:03:43.006: INFO: csi-mockplugin-resizer-0 started at 2023-01-22 16:00:55 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:43.006: INFO: 	Container csi-resizer ready: true, restart count 0
Jan 22 16:03:43.006: INFO: test-pod started at 2023-01-22 16:02:12 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:43.006: INFO: 	Container nginx ready: false, restart count 0
Jan 22 16:03:43.006: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-bzv42 started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:43.006: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:03:43.122: INFO: 
Latency metrics for node ip-172-20-45-186.us-east-2.compute.internal
Jan 22 16:03:43.122: INFO: 
Logging node info for node ip-172-20-52-12.us-east-2.compute.internal
Jan 22 16:03:43.153: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-52-12.us-east-2.compute.internal    33149587-e10e-48d5-abaf-ca8103e85c14 6683 0 2023-01-22 15:56:13 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-east-2 failure-domain.beta.kubernetes.io/zone:us-east-2a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-52-12.us-east-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/control-plane: node-role.kubernetes.io/master: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-east-2a topology.kubernetes.io/region:us-east-2 topology.kubernetes.io/zone:us-east-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0f06483805887653c"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kubelet Update v1 2023-01-22 15:56:13 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {protokube Update v1 2023-01-22 15:56:32 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kops-controller Update v1 2023-01-22 15:56:36 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/master":{}}}} } {kube-controller-manager Update v1 2023-01-22 15:56:54 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-01-22 15:57:53 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///us-east-2a/i-0f06483805887653c,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{47441653760 0} {<nil>} 46329740Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3895427072 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{42697488314 0} {<nil>} 42697488314 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3790569472 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-22 16:03:11 +0000 UTC,LastTransitionTime:2023-01-22 15:56:13 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-22 16:03:11 +0000 UTC,LastTransitionTime:2023-01-22 15:56:13 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-22 16:03:11 +0000 UTC,LastTransitionTime:2023-01-22 15:56:13 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-22 16:03:11 +0000 UTC,LastTransitionTime:2023-01-22 15:56:54 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.52.12,},NodeAddress{Type:ExternalIP,Address:3.16.137.247,},NodeAddress{Type:Hostname,Address:ip-172-20-52-12.us-east-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-52-12.us-east-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-16-137-247.us-east-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec221e40526c51c23560b6cf49b17a65,SystemUUID:ec221e40-526c-51c2-3560-b6cf49b17a65,BootID:89f5d760-653a-4491-8f94-5adaf8195cda,KernelVersion:5.15.86-flatcar,OSImage:Flatcar Container Linux by Kinvolk 3446.1.0 (Oklo),ContainerRuntimeVersion:docker://20.10.21,KubeletVersion:v1.23.16,KubeProxyVersion:v1.23.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:66a453db625abb268f4b3bbefc5a34a171d81e6e8796cecca54cfd71775c77c4 registry.k8s.io/etcdadm/etcd-manager:v3.0.20221209],SizeBytes:662196755,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456551323,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64:v1.23.16 registry.k8s.io/kube-apiserver-amd64:v1.23.16],SizeBytes:129993155,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64:v1.23.16 registry.k8s.io/kube-controller-manager-amd64:v1.23.16],SizeBytes:119933673,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.16 registry.k8s.io/kube-proxy-amd64:v1.23.16],SizeBytes:110829687,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:110016956,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:105636625,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:102694452,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:94469968,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:57729376,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:55458084,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:54841901,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64:v1.23.16 registry.k8s.io/kube-scheduler-amd64:v1.23.16],SizeBytes:51845852,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19575418,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17605286,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:8785817,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Jan 22 16:03:43.153: INFO: 
Logging kubelet events for node ip-172-20-52-12.us-east-2.compute.internal
Jan 22 16:03:43.184: INFO: 
Logging pods the kubelet thinks is on node ip-172-20-52-12.us-east-2.compute.internal
Jan 22 16:03:43.221: INFO: etcd-manager-main-ip-172-20-52-12.us-east-2.compute.internal started at 2023-01-22 15:55:26 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:43.221: INFO: 	Container etcd-manager ready: true, restart count 0
Jan 22 16:03:43.221: INFO: kops-controller-c9664 started at 2023-01-22 15:56:35 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:43.221: INFO: 	Container kops-controller ready: true, restart count 0
Jan 22 16:03:43.221: INFO: ebs-csi-node-288bv started at 2023-01-22 15:56:35 +0000 UTC (0+3 container statuses recorded)
Jan 22 16:03:43.221: INFO: 	Container ebs-plugin ready: true, restart count 0
Jan 22 16:03:43.221: INFO: 	Container liveness-probe ready: true, restart count 0
Jan 22 16:03:43.221: INFO: 	Container node-driver-registrar ready: true, restart count 0
Jan 22 16:03:43.221: INFO: cilium-operator-68bc649ff6-vvr4s started at 2023-01-22 15:56:35 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:43.221: INFO: 	Container cilium-operator ready: true, restart count 0
Jan 22 16:03:43.221: INFO: etcd-manager-cilium-ip-172-20-52-12.us-east-2.compute.internal started at 2023-01-22 15:55:26 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:43.221: INFO: 	Container etcd-manager ready: true, restart count 0
Jan 22 16:03:43.221: INFO: kube-apiserver-ip-172-20-52-12.us-east-2.compute.internal started at 2023-01-22 15:55:26 +0000 UTC (0+2 container statuses recorded)
Jan 22 16:03:43.221: INFO: 	Container healthcheck ready: true, restart count 0
Jan 22 16:03:43.221: INFO: 	Container kube-apiserver ready: true, restart count 1
Jan 22 16:03:43.221: INFO: kube-controller-manager-ip-172-20-52-12.us-east-2.compute.internal started at 2023-01-22 15:55:26 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:43.221: INFO: 	Container kube-controller-manager ready: true, restart count 2
Jan 22 16:03:43.221: INFO: kube-scheduler-ip-172-20-52-12.us-east-2.compute.internal started at 2023-01-22 15:55:26 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:43.221: INFO: 	Container kube-scheduler ready: true, restart count 0
Jan 22 16:03:43.221: INFO: cilium-tqjcp started at 2023-01-22 15:56:35 +0000 UTC (1+1 container statuses recorded)
Jan 22 16:03:43.221: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Jan 22 16:03:43.221: INFO: 	Container cilium-agent ready: true, restart count 0
Jan 22 16:03:43.221: INFO: ebs-csi-controller-54b65c69b6-ljh8m started at 2023-01-22 15:56:35 +0000 UTC (0+5 container statuses recorded)
Jan 22 16:03:43.221: INFO: 	Container csi-attacher ready: true, restart count 0
Jan 22 16:03:43.221: INFO: 	Container csi-provisioner ready: true, restart count 0
Jan 22 16:03:43.221: INFO: 	Container csi-resizer ready: true, restart count 0
Jan 22 16:03:43.221: INFO: 	Container ebs-plugin ready: true, restart count 0
Jan 22 16:03:43.221: INFO: 	Container liveness-probe ready: true, restart count 0
Jan 22 16:03:43.221: INFO: dns-controller-7db8b74b99-5rkk4 started at 2023-01-22 15:56:35 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:43.221: INFO: 	Container dns-controller ready: true, restart count 0
Jan 22 16:03:43.221: INFO: etcd-manager-events-ip-172-20-52-12.us-east-2.compute.internal started at 2023-01-22 15:55:26 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:03:43.221: INFO: 	Container etcd-manager ready: true, restart count 0
Jan 22 16:03:43.328: INFO: 
Latency metrics for node ip-172-20-52-12.us-east-2.compute.internal
Jan 22 16:03:43.328: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
�[1mSTEP�[0m: Destroying namespace "conntrack-2324" for this suite.

				Click to see stdout/stderrfrom junit_13.xml

Filter through log files

Kubernetes e2e suite [sig-storage] In-tree Volumes [Driver: hostPathSymlink] [Testpattern: Inline-volume (default fs)] subPath should support readOnly file specified in the volumeMount [LinuxOnly] 5m31s

go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[sig\-storage\]\sIn\-tree\sVolumes\s\[Driver\:\shostPathSymlink\]\s\[Testpattern\:\sInline\-volume\s\(default\sfs\)\]\ssubPath\sshould\ssupport\sreadOnly\sfile\sspecified\sin\sthe\svolumeMount\s\[LinuxOnly\]$'

/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:380
Jan 22 16:07:26.375: Unexpected error:
    <*errors.errorString | 0xc003726c60>: {
        s: "expected pod \"pod-subpath-test-inlinevolume-r9xz\" success: Gave up after waiting 5m0s for pod \"pod-subpath-test-inlinevolume-r9xz\" to be \"Succeeded or Failed\"",
    }
    expected pod "pod-subpath-test-inlinevolume-r9xz" success: Gave up after waiting 5m0s for pod "pod-subpath-test-inlinevolume-r9xz" to be "Succeeded or Failed"
occurred
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:770
				[BeforeEach] [Testpattern: Inline-volume (default fs)] subPath
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Inline-volume (default fs)] subPath
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185
�[1mSTEP�[0m: Creating a kubernetes client
Jan 22 16:02:12.961: INFO: >>> kubeConfig: /root/.kube/config
�[1mSTEP�[0m: Building a namespace api object, basename provisioning
�[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace
�[1mSTEP�[0m: Waiting for kube-root-ca.crt to be provisioned in namespace
[It] should support readOnly file specified in the volumeMount [LinuxOnly]
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:380
Jan 22 16:02:13.166: INFO: In-tree plugin kubernetes.io/host-path is not migrated, not validating any metrics
Jan 22 16:02:13.227: INFO: Waiting up to 5m0s for pod "hostpath-symlink-prep-provisioning-7201" in namespace "provisioning-7201" to be "Succeeded or Failed"
Jan 22 16:02:13.256: INFO: Pod "hostpath-symlink-prep-provisioning-7201": Phase="Pending", Reason="", readiness=false. Elapsed: 28.859264ms
Jan 22 16:02:15.286: INFO: Pod "hostpath-symlink-prep-provisioning-7201": Phase="Pending", Reason="", readiness=false. Elapsed: 2.059367722s
Jan 22 16:02:17.324: INFO: Pod "hostpath-symlink-prep-provisioning-7201": Phase="Pending", Reason="", readiness=false. Elapsed: 4.096654641s
Jan 22 16:02:19.353: INFO: Pod "hostpath-symlink-prep-provisioning-7201": Phase="Pending", Reason="", readiness=false. Elapsed: 6.126223587s
Jan 22 16:02:21.424: INFO: Pod "hostpath-symlink-prep-provisioning-7201": Phase="Succeeded", Reason="", readiness=false. Elapsed: 8.19653799s
�[1mSTEP�[0m: Saw pod success
Jan 22 16:02:21.424: INFO: Pod "hostpath-symlink-prep-provisioning-7201" satisfied condition "Succeeded or Failed"
Jan 22 16:02:21.424: INFO: Deleting pod "hostpath-symlink-prep-provisioning-7201" in namespace "provisioning-7201"
Jan 22 16:02:21.507: INFO: Wait up to 5m0s for pod "hostpath-symlink-prep-provisioning-7201" to be fully deleted
Jan 22 16:02:21.542: INFO: Creating resource for inline volume
�[1mSTEP�[0m: Creating pod pod-subpath-test-inlinevolume-r9xz
�[1mSTEP�[0m: Creating a pod to test subpath
Jan 22 16:02:21.591: INFO: Waiting up to 5m0s for pod "pod-subpath-test-inlinevolume-r9xz" in namespace "provisioning-7201" to be "Succeeded or Failed"
Jan 22 16:02:21.633: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 41.897502ms
Jan 22 16:02:23.664: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 2.072710402s
Jan 22 16:02:25.694: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 4.1023582s
Jan 22 16:02:27.725: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 6.133552894s
Jan 22 16:02:29.754: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 8.163020475s
Jan 22 16:02:31.786: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 10.194135668s
Jan 22 16:02:33.816: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 12.224705589s
Jan 22 16:02:35.846: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 14.254253423s
Jan 22 16:02:37.876: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 16.284053161s
Jan 22 16:02:39.906: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 18.314984355s
Jan 22 16:02:41.938: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 20.346069818s
Jan 22 16:02:43.967: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 22.375973798s
Jan 22 16:02:45.997: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 24.405446537s
Jan 22 16:02:48.027: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 26.435128331s
Jan 22 16:02:50.057: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 28.465474954s
Jan 22 16:02:52.088: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 30.4964897s
Jan 22 16:02:54.119: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 32.527225775s
Jan 22 16:02:56.149: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 34.55788348s
Jan 22 16:02:58.179: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 36.587944339s
Jan 22 16:03:00.212: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 38.620390288s
Jan 22 16:03:02.243: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 40.651057448s
Jan 22 16:03:04.272: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 42.680843389s
Jan 22 16:03:06.302: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 44.710175859s
Jan 22 16:03:08.331: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 46.739802002s
Jan 22 16:03:10.362: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 48.770471521s
Jan 22 16:03:12.403: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 50.811282764s
Jan 22 16:03:14.434: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 52.842424215s
Jan 22 16:03:16.468: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 54.876102131s
Jan 22 16:03:18.497: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 56.905655134s
Jan 22 16:03:20.528: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 58.936682372s
Jan 22 16:03:22.559: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 1m0.967175252s
Jan 22 16:03:24.591: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 1m2.99944509s
Jan 22 16:03:26.620: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 1m5.028844386s
Jan 22 16:03:28.650: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 1m7.058648377s
Jan 22 16:03:30.681: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 1m9.08961148s
Jan 22 16:03:32.711: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 1m11.119697431s
Jan 22 16:03:34.744: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 1m13.152909645s
Jan 22 16:03:36.775: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 1m15.183493414s
Jan 22 16:03:38.805: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 1m17.213478445s
Jan 22 16:03:40.836: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 1m19.244172797s
Jan 22 16:03:42.866: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 1m21.274971828s
Jan 22 16:03:44.899: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 1m23.30759379s
Jan 22 16:03:46.929: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 1m25.337508219s
Jan 22 16:03:48.960: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 1m27.368164223s
Jan 22 16:03:50.991: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 1m29.399332237s
Jan 22 16:03:53.021: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 1m31.429772235s
Jan 22 16:03:55.052: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 1m33.461015437s
Jan 22 16:03:57.082: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 1m35.491013268s
Jan 22 16:03:59.113: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 1m37.521148966s
Jan 22 16:04:01.142: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 1m39.550715663s
Jan 22 16:04:03.172: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 1m41.58018558s
Jan 22 16:04:05.203: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 1m43.611735333s
Jan 22 16:04:07.233: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 1m45.641632209s
Jan 22 16:04:09.264: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 1m47.672443128s
Jan 22 16:04:11.294: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 1m49.702308886s
Jan 22 16:04:13.324: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 1m51.732300087s
Jan 22 16:04:15.354: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 1m53.762277576s
Jan 22 16:04:17.384: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 1m55.792492208s
Jan 22 16:04:19.414: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 1m57.822101306s
Jan 22 16:04:21.443: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 1m59.85173698s
Jan 22 16:04:23.473: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 2m1.881411363s
Jan 22 16:04:25.502: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 2m3.910830345s
Jan 22 16:04:27.533: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 2m5.941229573s
Jan 22 16:04:29.563: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 2m7.971892203s
Jan 22 16:04:31.594: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 2m10.002566261s
Jan 22 16:04:33.623: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 2m12.031986462s
Jan 22 16:04:35.656: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 2m14.064400555s
Jan 22 16:04:37.686: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 2m16.094696624s
Jan 22 16:04:39.716: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 2m18.124459171s
Jan 22 16:04:41.746: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 2m20.15485377s
Jan 22 16:04:43.776: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 2m22.184739948s
Jan 22 16:04:45.808: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 2m24.216434071s
Jan 22 16:04:47.837: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 2m26.245912381s
Jan 22 16:04:49.868: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 2m28.276500572s
Jan 22 16:04:51.899: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 2m30.307376564s
Jan 22 16:04:53.931: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 2m32.339964134s
Jan 22 16:04:55.963: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 2m34.371267389s
Jan 22 16:04:57.995: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 2m36.403611465s
Jan 22 16:05:00.026: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 2m38.434261364s
Jan 22 16:05:02.056: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 2m40.464146794s
Jan 22 16:05:04.088: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 2m42.496198467s
Jan 22 16:05:06.118: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 2m44.526966866s
Jan 22 16:05:08.150: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 2m46.558794505s
Jan 22 16:05:10.180: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 2m48.588745129s
Jan 22 16:05:12.212: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 2m50.620864027s
Jan 22 16:05:14.242: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 2m52.650932123s
Jan 22 16:05:16.273: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 2m54.68153719s
Jan 22 16:05:18.303: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 2m56.711611335s
Jan 22 16:05:20.334: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 2m58.742386484s
Jan 22 16:05:22.364: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 3m0.77231757s
Jan 22 16:05:24.395: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 3m2.803647866s
Jan 22 16:05:26.427: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 3m4.835359783s
Jan 22 16:05:28.456: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 3m6.865043681s
Jan 22 16:05:30.487: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 3m8.895873151s
Jan 22 16:05:32.518: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 3m10.926285878s
Jan 22 16:05:34.548: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 3m12.956527614s
Jan 22 16:05:36.579: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 3m14.987767936s
Jan 22 16:05:38.609: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 3m17.017501386s
Jan 22 16:05:40.639: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 3m19.048006643s
Jan 22 16:05:42.670: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 3m21.078688244s
Jan 22 16:05:44.700: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 3m23.108402748s
Jan 22 16:05:46.730: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 3m25.138741331s
Jan 22 16:05:48.760: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 3m27.168412731s
Jan 22 16:05:50.791: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 3m29.199113041s
Jan 22 16:05:52.820: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 3m31.228783916s
Jan 22 16:05:54.850: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 3m33.259025354s
Jan 22 16:05:56.880: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 3m35.288662554s
Jan 22 16:05:58.910: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 3m37.318824235s
Jan 22 16:06:00.942: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 3m39.350247428s
Jan 22 16:06:02.972: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 3m41.380533361s
Jan 22 16:06:05.006: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 3m43.414127938s
Jan 22 16:06:07.036: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 3m45.444870253s
Jan 22 16:06:09.066: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 3m47.474959624s
Jan 22 16:06:11.097: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 3m49.505960431s
Jan 22 16:06:13.128: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 3m51.53608979s
Jan 22 16:06:15.157: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 3m53.565789847s
Jan 22 16:06:17.188: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 3m55.596804809s
Jan 22 16:06:19.218: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 3m57.626470121s
Jan 22 16:06:21.248: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 3m59.656523099s
Jan 22 16:06:23.278: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 4m1.687025827s
Jan 22 16:06:25.309: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 4m3.717578431s
Jan 22 16:06:27.339: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 4m5.747319057s
Jan 22 16:06:29.368: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 4m7.776980125s
Jan 22 16:06:31.398: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 4m9.806883072s
Jan 22 16:06:33.428: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 4m11.836744652s
Jan 22 16:06:35.458: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 4m13.866510736s
Jan 22 16:06:37.488: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 4m15.896905542s
Jan 22 16:06:39.518: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 4m17.926544913s
Jan 22 16:06:41.554: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 4m19.962947949s
Jan 22 16:06:43.584: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 4m21.992480017s
Jan 22 16:06:45.615: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 4m24.023531512s
Jan 22 16:06:47.647: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 4m26.055915515s
Jan 22 16:06:49.678: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 4m28.086221596s
Jan 22 16:06:51.708: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 4m30.11660115s
Jan 22 16:06:53.738: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 4m32.146965309s
Jan 22 16:06:55.768: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 4m34.177033874s
Jan 22 16:06:57.799: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 4m36.207501707s
Jan 22 16:06:59.828: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 4m38.236948244s
Jan 22 16:07:01.859: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 4m40.267949154s
Jan 22 16:07:03.889: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 4m42.297516818s
Jan 22 16:07:05.920: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 4m44.32820669s
Jan 22 16:07:07.957: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 4m46.365279751s
Jan 22 16:07:09.989: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 4m48.397235448s
Jan 22 16:07:12.020: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 4m50.428049122s
Jan 22 16:07:14.049: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 4m52.457911742s
Jan 22 16:07:16.082: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 4m54.490329125s
Jan 22 16:07:18.111: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 4m56.519989866s
Jan 22 16:07:20.142: INFO: Pod "pod-subpath-test-inlinevolume-r9xz": Phase="Pending", Reason="", readiness=false. Elapsed: 4m58.550565374s
Jan 22 16:07:22.213: INFO: Output of node "ip-172-20-39-104.us-east-2.compute.internal" pod "pod-subpath-test-inlinevolume-r9xz" container "init-volume-inlinevolume-r9xz": 
Jan 22 16:07:22.247: INFO: Output of node "ip-172-20-39-104.us-east-2.compute.internal" pod "pod-subpath-test-inlinevolume-r9xz" container "test-init-volume-inlinevolume-r9xz": content of file "/test-volume/provisioning-7201": mount-tester new file

mode of file "/test-volume/provisioning-7201": -rw-r--r--

Jan 22 16:07:22.282: INFO: Output of node "ip-172-20-39-104.us-east-2.compute.internal" pod "pod-subpath-test-inlinevolume-r9xz" container "test-container-subpath-inlinevolume-r9xz": content of file "/test-volume": mount-tester new file


�[1mSTEP�[0m: delete the pod
Jan 22 16:07:22.314: INFO: Waiting for pod pod-subpath-test-inlinevolume-r9xz to disappear
Jan 22 16:07:22.344: INFO: Pod pod-subpath-test-inlinevolume-r9xz still exists
Jan 22 16:07:24.344: INFO: Waiting for pod pod-subpath-test-inlinevolume-r9xz to disappear
Jan 22 16:07:24.374: INFO: Pod pod-subpath-test-inlinevolume-r9xz still exists
Jan 22 16:07:26.346: INFO: Waiting for pod pod-subpath-test-inlinevolume-r9xz to disappear
Jan 22 16:07:26.375: INFO: Pod pod-subpath-test-inlinevolume-r9xz no longer exists
Jan 22 16:07:26.375: FAIL: Unexpected error:
    <*errors.errorString | 0xc003726c60>: {
        s: "expected pod \"pod-subpath-test-inlinevolume-r9xz\" success: Gave up after waiting 5m0s for pod \"pod-subpath-test-inlinevolume-r9xz\" to be \"Succeeded or Failed\"",
    }
    expected pod "pod-subpath-test-inlinevolume-r9xz" success: Gave up after waiting 5m0s for pod "pod-subpath-test-inlinevolume-r9xz" to be "Succeeded or Failed"
occurred

Full Stack Trace
k8s.io/kubernetes/test/e2e/framework.(*Framework).testContainerOutputMatcher(0xc004ce3c00?, {0x705074c?, 0x0?}, 0xc004ce3c00, 0x0, {0xc0038bf118, 0x1, 0x1}, 0x2a?)
	/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:770 +0x176
k8s.io/kubernetes/test/e2e/framework.(*Framework).TestContainerOutput(...)
	/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:567
k8s.io/kubernetes/test/e2e/storage/testsuites.testReadFile(0xc002594b00, {0x705e233, 0xc}, 0xc004ce3c00, 0x0)
	/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:691 +0x15c
k8s.io/kubernetes/test/e2e/storage/testsuites.(*subPathTestSuite).DefineTests.func17()
	/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:392 +0x1e5
k8s.io/kubernetes/test/e2e.RunE2ETests(0x24eec17?)
	_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:133 +0x6b7
k8s.io/kubernetes/test/e2e.TestE2E(0x24602d9?)
	_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:136 +0x19
testing.tRunner(0xc00093c9c0, 0x72ecb90)
	/usr/local/go/src/testing/testing.go:1446 +0x10b
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:1493 +0x35f
�[1mSTEP�[0m: Deleting pod
Jan 22 16:07:26.376: INFO: Deleting pod "pod-subpath-test-inlinevolume-r9xz" in namespace "provisioning-7201"
Jan 22 16:07:26.441: INFO: Waiting up to 5m0s for pod "hostpath-symlink-prep-provisioning-7201" in namespace "provisioning-7201" to be "Succeeded or Failed"
Jan 22 16:07:26.469: INFO: Pod "hostpath-symlink-prep-provisioning-7201": Phase="Pending", Reason="", readiness=false. Elapsed: 28.734879ms
Jan 22 16:07:28.499: INFO: Pod "hostpath-symlink-prep-provisioning-7201": Phase="Pending", Reason="", readiness=false. Elapsed: 2.058279319s
Jan 22 16:07:30.531: INFO: Pod "hostpath-symlink-prep-provisioning-7201": Phase="Pending", Reason="", readiness=false. Elapsed: 4.090715066s
Jan 22 16:07:32.568: INFO: Pod "hostpath-symlink-prep-provisioning-7201": Phase="Pending", Reason="", readiness=false. Elapsed: 6.127392429s
Jan 22 16:07:34.599: INFO: Pod "hostpath-symlink-prep-provisioning-7201": Phase="Pending", Reason="", readiness=false. Elapsed: 8.157933596s
Jan 22 16:07:36.629: INFO: Pod "hostpath-symlink-prep-provisioning-7201": Phase="Pending", Reason="", readiness=false. Elapsed: 10.188456764s
Jan 22 16:07:38.659: INFO: Pod "hostpath-symlink-prep-provisioning-7201": Phase="Pending", Reason="", readiness=false. Elapsed: 12.217902808s
Jan 22 16:07:40.689: INFO: Pod "hostpath-symlink-prep-provisioning-7201": Phase="Pending", Reason="", readiness=false. Elapsed: 14.248428822s
Jan 22 16:07:42.719: INFO: Pod "hostpath-symlink-prep-provisioning-7201": Phase="Succeeded", Reason="", readiness=false. Elapsed: 16.278464102s
�[1mSTEP�[0m: Saw pod success
Jan 22 16:07:42.719: INFO: Pod "hostpath-symlink-prep-provisioning-7201" satisfied condition "Succeeded or Failed"
Jan 22 16:07:42.719: INFO: Deleting pod "hostpath-symlink-prep-provisioning-7201" in namespace "provisioning-7201"
Jan 22 16:07:42.755: INFO: Wait up to 5m0s for pod "hostpath-symlink-prep-provisioning-7201" to be fully deleted
[AfterEach] [Testpattern: Inline-volume (default fs)] subPath
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186
�[1mSTEP�[0m: Collecting events from namespace "provisioning-7201".
�[1mSTEP�[0m: Found 22 events.
Jan 22 16:07:42.815: INFO: At 2023-01-22 16:02:15 +0000 UTC - event for hostpath-symlink-prep-provisioning-7201: {kubelet ip-172-20-39-104.us-east-2.compute.internal} Created: Created container init-volume-provisioning-7201
Jan 22 16:07:42.815: INFO: At 2023-01-22 16:02:15 +0000 UTC - event for hostpath-symlink-prep-provisioning-7201: {kubelet ip-172-20-39-104.us-east-2.compute.internal} Started: Started container init-volume-provisioning-7201
Jan 22 16:07:42.815: INFO: At 2023-01-22 16:02:15 +0000 UTC - event for hostpath-symlink-prep-provisioning-7201: {kubelet ip-172-20-39-104.us-east-2.compute.internal} Pulled: Container image "k8s.gcr.io/e2e-test-images/busybox:1.29-2" already present on machine
Jan 22 16:07:42.815: INFO: At 2023-01-22 16:02:21 +0000 UTC - event for pod-subpath-test-inlinevolume-r9xz: {default-scheduler } Scheduled: Successfully assigned provisioning-7201/pod-subpath-test-inlinevolume-r9xz to ip-172-20-39-104.us-east-2.compute.internal
Jan 22 16:07:42.815: INFO: At 2023-01-22 16:03:23 +0000 UTC - event for pod-subpath-test-inlinevolume-r9xz: {kubelet ip-172-20-39-104.us-east-2.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "eaced20bad820c5e0be8ec1b86fd018470fa23818a414ed93c7d392d6a75d168" network for pod "pod-subpath-test-inlinevolume-r9xz": networkPlugin cni failed to set up pod "pod-subpath-test-inlinevolume-r9xz_provisioning-7201" network: unable to connect to Cilium daemon: failed to create cilium agent client after 30.000000 seconds timeout: Get "http:///var/run/cilium/cilium.sock/v1/config": dial unix /var/run/cilium/cilium.sock: connect: no such file or directory
Is the agent running?
Jan 22 16:07:42.815: INFO: At 2023-01-22 16:03:23 +0000 UTC - event for pod-subpath-test-inlinevolume-r9xz: {kubelet ip-172-20-39-104.us-east-2.compute.internal} SandboxChanged: Pod sandbox changed, it will be killed and re-created.
Jan 22 16:07:42.815: INFO: At 2023-01-22 16:04:54 +0000 UTC - event for pod-subpath-test-inlinevolume-r9xz: {kubelet ip-172-20-39-104.us-east-2.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "83407a2fcb335920c6d6e76eb029a2e7a3d19372afbf66da5c138a3c26a7cb10" network for pod "pod-subpath-test-inlinevolume-r9xz": networkPlugin cni failed to set up pod "pod-subpath-test-inlinevolume-r9xz_provisioning-7201" network: unable to connect to Cilium daemon: failed to create cilium agent client after 30.000000 seconds timeout: Get "http:///var/run/cilium/cilium.sock/v1/config": dial unix /var/run/cilium/cilium.sock: connect: no such file or directory
Is the agent running?
Jan 22 16:07:42.815: INFO: At 2023-01-22 16:06:08 +0000 UTC - event for pod-subpath-test-inlinevolume-r9xz: {kubelet ip-172-20-39-104.us-east-2.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "4815462f100581eec3e534c6093009dd7c83d755f87ff84ccefa09c63d11cbae" network for pod "pod-subpath-test-inlinevolume-r9xz": networkPlugin cni failed to set up pod "pod-subpath-test-inlinevolume-r9xz_provisioning-7201" network: unable to connect to Cilium daemon: failed to create cilium agent client after 30.000000 seconds timeout: Get "http:///var/run/cilium/cilium.sock/v1/config": dial unix /var/run/cilium/cilium.sock: connect: no such file or directory
Is the agent running?
Jan 22 16:07:42.815: INFO: At 2023-01-22 16:06:23 +0000 UTC - event for pod-subpath-test-inlinevolume-r9xz: {kubelet ip-172-20-39-104.us-east-2.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "445bec5fc5b84451059a5c95ae206cc510fa2b9687516537ad4c6a69c4103bfa" network for pod "pod-subpath-test-inlinevolume-r9xz": networkPlugin cni failed to set up pod "pod-subpath-test-inlinevolume-r9xz_provisioning-7201" network: Unable to create endpoint: [PUT /endpoint/{id}][429] putEndpointIdTooManyRequests 
Jan 22 16:07:42.815: INFO: At 2023-01-22 16:06:51 +0000 UTC - event for pod-subpath-test-inlinevolume-r9xz: {kubelet ip-172-20-39-104.us-east-2.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "d9169486f0a7fee2616c8a42be5221a65070392bf6d2e09a00e3cff58f2815d2" network for pod "pod-subpath-test-inlinevolume-r9xz": networkPlugin cni failed to set up pod "pod-subpath-test-inlinevolume-r9xz_provisioning-7201" network: Unable to create endpoint: [PUT /endpoint/{id}][429] putEndpointIdTooManyRequests 
Jan 22 16:07:42.815: INFO: At 2023-01-22 16:07:05 +0000 UTC - event for pod-subpath-test-inlinevolume-r9xz: {kubelet ip-172-20-39-104.us-east-2.compute.internal} Created: Created container init-volume-inlinevolume-r9xz
Jan 22 16:07:42.815: INFO: At 2023-01-22 16:07:05 +0000 UTC - event for pod-subpath-test-inlinevolume-r9xz: {kubelet ip-172-20-39-104.us-east-2.compute.internal} Pulled: Container image "k8s.gcr.io/e2e-test-images/busybox:1.29-2" already present on machine
Jan 22 16:07:42.815: INFO: At 2023-01-22 16:07:06 +0000 UTC - event for pod-subpath-test-inlinevolume-r9xz: {kubelet ip-172-20-39-104.us-east-2.compute.internal} Started: Started container init-volume-inlinevolume-r9xz
Jan 22 16:07:42.815: INFO: At 2023-01-22 16:07:06 +0000 UTC - event for pod-subpath-test-inlinevolume-r9xz: {kubelet ip-172-20-39-104.us-east-2.compute.internal} Pulled: Container image "k8s.gcr.io/e2e-test-images/agnhost:2.39" already present on machine
Jan 22 16:07:42.815: INFO: At 2023-01-22 16:07:06 +0000 UTC - event for pod-subpath-test-inlinevolume-r9xz: {kubelet ip-172-20-39-104.us-east-2.compute.internal} Created: Created container test-init-volume-inlinevolume-r9xz
Jan 22 16:07:42.815: INFO: At 2023-01-22 16:07:07 +0000 UTC - event for pod-subpath-test-inlinevolume-r9xz: {kubelet ip-172-20-39-104.us-east-2.compute.internal} Started: Started container test-init-volume-inlinevolume-r9xz
Jan 22 16:07:42.815: INFO: At 2023-01-22 16:07:07 +0000 UTC - event for pod-subpath-test-inlinevolume-r9xz: {kubelet ip-172-20-39-104.us-east-2.compute.internal} Pulled: Container image "k8s.gcr.io/e2e-test-images/agnhost:2.39" already present on machine
Jan 22 16:07:42.815: INFO: At 2023-01-22 16:07:07 +0000 UTC - event for pod-subpath-test-inlinevolume-r9xz: {kubelet ip-172-20-39-104.us-east-2.compute.internal} Created: Created container test-container-subpath-inlinevolume-r9xz
Jan 22 16:07:42.815: INFO: At 2023-01-22 16:07:08 +0000 UTC - event for pod-subpath-test-inlinevolume-r9xz: {kubelet ip-172-20-39-104.us-east-2.compute.internal} Started: Started container test-container-subpath-inlinevolume-r9xz
Jan 22 16:07:42.815: INFO: At 2023-01-22 16:07:28 +0000 UTC - event for hostpath-symlink-prep-provisioning-7201: {kubelet ip-172-20-39-104.us-east-2.compute.internal} Started: Started container init-volume-provisioning-7201
Jan 22 16:07:42.815: INFO: At 2023-01-22 16:07:28 +0000 UTC - event for hostpath-symlink-prep-provisioning-7201: {kubelet ip-172-20-39-104.us-east-2.compute.internal} Created: Created container init-volume-provisioning-7201
Jan 22 16:07:42.815: INFO: At 2023-01-22 16:07:28 +0000 UTC - event for hostpath-symlink-prep-provisioning-7201: {kubelet ip-172-20-39-104.us-east-2.compute.internal} Pulled: Container image "k8s.gcr.io/e2e-test-images/busybox:1.29-2" already present on machine
Jan 22 16:07:42.844: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
Jan 22 16:07:42.844: INFO: 
Jan 22 16:07:42.874: INFO: 
Logging node info for node ip-172-20-39-104.us-east-2.compute.internal
Jan 22 16:07:42.903: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-39-104.us-east-2.compute.internal    94a127fa-2d03-4fae-878f-95cc778a7072 12613 0 2023-01-22 15:57:23 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-east-2 failure-domain.beta.kubernetes.io/zone:us-east-2a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-39-104.us-east-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-east-2a topology.hostpath.csi/node:ip-172-20-39-104.us-east-2.compute.internal topology.kubernetes.io/region:us-east-2 topology.kubernetes.io/zone:us-east-2a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-339":"csi-mock-csi-mock-volumes-339","csi-mock-csi-mock-volumes-4044":"csi-mock-csi-mock-volumes-4044","ebs.csi.aws.com":"i-058f72b2ec21c63a8"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kops-controller Update v1 2023-01-22 15:57:23 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-22 15:57:23 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {kubelet Update v1 2023-01-22 15:57:23 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2023-01-22 16:02:09 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-22 16:02:16 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status} {e2e.test Update v1 2023-01-22 16:02:30 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} }]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///us-east-2a/i-058f72b2ec21c63a8,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{47441653760 0} {<nil>} 46329740Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4054806528 0} {<nil>} 3959772Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{42697488314 0} {<nil>} 42697488314 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3949948928 0} {<nil>} 3857372Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-22 16:07:39 +0000 UTC,LastTransitionTime:2023-01-22 15:57:23 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-22 16:07:39 +0000 UTC,LastTransitionTime:2023-01-22 15:57:23 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-22 16:07:39 +0000 UTC,LastTransitionTime:2023-01-22 15:57:23 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-22 16:07:39 +0000 UTC,LastTransitionTime:2023-01-22 15:57:44 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.39.104,},NodeAddress{Type:ExternalIP,Address:3.133.79.156,},NodeAddress{Type:Hostname,Address:ip-172-20-39-104.us-east-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-39-104.us-east-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-133-79-156.us-east-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2455e84fc0e92553e5c201c3d73752,SystemUUID:ec2455e8-4fc0-e925-53e5-c201c3d73752,BootID:baa49577-2567-45b4-b63f-1f65be12643c,KernelVersion:5.15.86-flatcar,OSImage:Flatcar Container Linux by Kinvolk 3446.1.0 (Oklo),ContainerRuntimeVersion:docker://20.10.21,KubeletVersion:v1.23.16,KubeProxyVersion:v1.23.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456551323,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126853136,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:99c0d6f1ad24a1aa1905d9c6534d193f268f7b23f9add2ae6bb41f31094bdd5c k8s.gcr.io/e2e-test-images/nautilus:1.5],SizeBytes:124738681,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123774598,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.16 registry.k8s.io/kube-proxy-amd64:v1.23.16],SizeBytes:110829687,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:94469968,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56366111,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56363051,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53940090,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53515772,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53484608,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53483352,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:50249337,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:49160266,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:47903915,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29972082,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19646681,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19575418,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18683218,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17664775,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17605286,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17159270,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16028126,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-00f9ca3f8a10b690b],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-00f9ca3f8a10b690b,DevicePath:,},},Config:nil,},}
Jan 22 16:07:42.904: INFO: 
Logging kubelet events for node ip-172-20-39-104.us-east-2.compute.internal
Jan 22 16:07:42.935: INFO: 
Logging pods the kubelet thinks is on node ip-172-20-39-104.us-east-2.compute.internal
Jan 22 16:07:43.008: INFO: pod-subpath-test-preprovisionedpv-96vn started at 2023-01-22 16:06:50 +0000 UTC (1+1 container statuses recorded)
Jan 22 16:07:43.008: INFO: 	Init container init-volume-preprovisionedpv-96vn ready: true, restart count 0
Jan 22 16:07:43.008: INFO: 	Container test-container-subpath-preprovisionedpv-96vn ready: false, restart count 0
Jan 22 16:07:43.008: INFO: csi-mockplugin-0 started at 2023-01-22 16:03:27 +0000 UTC (0+3 container statuses recorded)
Jan 22 16:07:43.008: INFO: 	Container csi-provisioner ready: true, restart count 0
Jan 22 16:07:43.008: INFO: 	Container driver-registrar ready: true, restart count 0
Jan 22 16:07:43.008: INFO: 	Container mock ready: true, restart count 0
Jan 22 16:07:43.008: INFO: ss2-2 started at <nil> (0+0 container statuses recorded)
Jan 22 16:07:43.008: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-7sknq started at 2023-01-22 16:02:31 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.009: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:07:43.009: INFO: csi-mockplugin-attacher-0 started at 2023-01-22 16:06:44 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.009: INFO: 	Container csi-attacher ready: true, restart count 0
Jan 22 16:07:43.009: INFO: cilium-cl25d started at 2023-01-22 15:57:24 +0000 UTC (1+1 container statuses recorded)
Jan 22 16:07:43.009: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Jan 22 16:07:43.009: INFO: 	Container cilium-agent ready: true, restart count 2
Jan 22 16:07:43.009: INFO: netserver-0 started at 2023-01-22 16:03:48 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.009: INFO: 	Container webserver ready: true, restart count 0
Jan 22 16:07:43.009: INFO: local-client started at 2023-01-22 16:07:31 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.009: INFO: 	Container local-client ready: false, restart count 0
Jan 22 16:07:43.009: INFO: ebs-csi-node-8fr2s started at 2023-01-22 15:57:24 +0000 UTC (0+3 container statuses recorded)
Jan 22 16:07:43.009: INFO: 	Container ebs-plugin ready: true, restart count 0
Jan 22 16:07:43.009: INFO: 	Container liveness-probe ready: true, restart count 0
Jan 22 16:07:43.009: INFO: 	Container node-driver-registrar ready: true, restart count 0
Jan 22 16:07:43.009: INFO: csi-mockplugin-attacher-0 started at 2023-01-22 16:02:37 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.009: INFO: 	Container csi-attacher ready: true, restart count 0
Jan 22 16:07:43.009: INFO: csi-mockplugin-resizer-0 started at 2023-01-22 16:02:37 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.009: INFO: 	Container csi-resizer ready: true, restart count 0
Jan 22 16:07:43.009: INFO: busybox-c1850765-faac-4c06-871f-90efbaeb9304 started at 2023-01-22 16:04:46 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.009: INFO: 	Container busybox ready: false, restart count 0
Jan 22 16:07:43.009: INFO: test-rolling-update-with-lb-58c5c55949-ptrmp started at 2023-01-22 16:07:24 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.009: INFO: 	Container agnhost ready: true, restart count 0
Jan 22 16:07:43.009: INFO: pfpod started at 2023-01-22 16:04:22 +0000 UTC (0+2 container statuses recorded)
Jan 22 16:07:43.009: INFO: 	Container portforwardtester ready: false, restart count 0
Jan 22 16:07:43.009: INFO: 	Container readiness ready: false, restart count 0
Jan 22 16:07:43.009: INFO: coredns-867df8f45c-7d727 started at 2023-01-22 15:57:44 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.009: INFO: 	Container coredns ready: true, restart count 0
Jan 22 16:07:43.009: INFO: hostexec-ip-172-20-39-104.us-east-2.compute.internal-4fczj started at 2023-01-22 16:06:52 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.009: INFO: 	Container agnhost-container ready: true, restart count 0
Jan 22 16:07:43.009: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-zltrj started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.009: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:07:43.009: INFO: hostexec-ip-172-20-39-104.us-east-2.compute.internal-zqlvd started at 2023-01-22 16:06:42 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.009: INFO: 	Container agnhost-container ready: true, restart count 0
Jan 22 16:07:43.009: INFO: ss-1 started at 2023-01-22 16:02:19 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.009: INFO: 	Container webserver ready: true, restart count 0
Jan 22 16:07:43.009: INFO: csi-mockplugin-0 started at 2023-01-22 16:06:44 +0000 UTC (0+3 container statuses recorded)
Jan 22 16:07:43.009: INFO: 	Container csi-provisioner ready: true, restart count 0
Jan 22 16:07:43.009: INFO: 	Container driver-registrar ready: true, restart count 0
Jan 22 16:07:43.009: INFO: 	Container mock ready: true, restart count 0
Jan 22 16:07:43.009: INFO: coredns-autoscaler-557ccb4c66-44k6v started at 2023-01-22 15:57:44 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.009: INFO: 	Container autoscaler ready: true, restart count 0
Jan 22 16:07:43.009: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-ltqph started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.009: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:07:43.009: INFO: csi-mockplugin-0 started at 2023-01-22 16:02:37 +0000 UTC (0+3 container statuses recorded)
Jan 22 16:07:43.009: INFO: 	Container csi-provisioner ready: true, restart count 0
Jan 22 16:07:43.009: INFO: 	Container driver-registrar ready: true, restart count 0
Jan 22 16:07:43.009: INFO: 	Container mock ready: true, restart count 0
Jan 22 16:07:43.009: INFO: netserver-0 started at 2023-01-22 16:03:41 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.009: INFO: 	Container webserver ready: true, restart count 0
Jan 22 16:07:43.128: INFO: 
Latency metrics for node ip-172-20-39-104.us-east-2.compute.internal
Jan 22 16:07:43.128: INFO: 
Logging node info for node ip-172-20-39-82.us-east-2.compute.internal
Jan 22 16:07:43.158: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-39-82.us-east-2.compute.internal    fab1195f-32aa-4044-ac45-046bbfde063a 11126 0 2023-01-22 15:57:34 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-east-2 failure-domain.beta.kubernetes.io/zone:us-east-2a io.kubernetes.storage.mock/node:some-mock-node kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-39-82.us-east-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-east-2a topology.kubernetes.io/region:us-east-2 topology.kubernetes.io/zone:us-east-2a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-9552":"csi-mock-csi-mock-volumes-9552","ebs.csi.aws.com":"i-0a00ce72536f23489"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kops-controller Update v1 2023-01-22 15:57:34 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-22 15:57:34 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2023-01-22 15:57:34 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2023-01-22 16:01:40 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {e2e.test Update v1 2023-01-22 16:02:30 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kubelet Update v1 2023-01-22 16:03:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///us-east-2a/i-0a00ce72536f23489,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{47441653760 0} {<nil>} 46329740Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4054806528 0} {<nil>} 3959772Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{42697488314 0} {<nil>} 42697488314 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3949948928 0} {<nil>} 3857372Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-22 16:07:06 +0000 UTC,LastTransitionTime:2023-01-22 15:57:34 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-22 16:07:06 +0000 UTC,LastTransitionTime:2023-01-22 15:57:34 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-22 16:07:06 +0000 UTC,LastTransitionTime:2023-01-22 15:57:34 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-22 16:07:06 +0000 UTC,LastTransitionTime:2023-01-22 15:57:54 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.39.82,},NodeAddress{Type:ExternalIP,Address:3.14.247.52,},NodeAddress{Type:Hostname,Address:ip-172-20-39-82.us-east-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-39-82.us-east-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-14-247-52.us-east-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bb4ac3588554994a702eabb6be703,SystemUUID:ec2bb4ac-3588-5549-94a7-02eabb6be703,BootID:82a481e4-e615-4371-bbc2-0b7f74e99966,KernelVersion:5.15.86-flatcar,OSImage:Flatcar Container Linux by Kinvolk 3446.1.0 (Oklo),ContainerRuntimeVersion:docker://20.10.21,KubeletVersion:v1.23.16,KubeProxyVersion:v1.23.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456551323,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:253346057,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126853136,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:99c0d6f1ad24a1aa1905d9c6534d193f268f7b23f9add2ae6bb41f31094bdd5c k8s.gcr.io/e2e-test-images/nautilus:1.5],SizeBytes:124738681,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123774598,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.16 registry.k8s.io/kube-proxy-amd64:v1.23.16],SizeBytes:110829687,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:94469968,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56366111,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53484608,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:50249337,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:49160266,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19646681,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19575418,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17664775,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17605286,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16028126,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-07558a7d8ec7e2f44],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-07558a7d8ec7e2f44,DevicePath:,},},Config:nil,},}
Jan 22 16:07:43.158: INFO: 
Logging kubelet events for node ip-172-20-39-82.us-east-2.compute.internal
Jan 22 16:07:43.189: INFO: 
Logging pods the kubelet thinks is on node ip-172-20-39-82.us-east-2.compute.internal
Jan 22 16:07:43.225: INFO: csi-mockplugin-0 started at 2023-01-22 16:06:47 +0000 UTC (0+3 container statuses recorded)
Jan 22 16:07:43.225: INFO: 	Container csi-provisioner ready: true, restart count 0
Jan 22 16:07:43.225: INFO: 	Container driver-registrar ready: true, restart count 0
Jan 22 16:07:43.225: INFO: 	Container mock ready: true, restart count 0
Jan 22 16:07:43.225: INFO: netserver-1 started at 2023-01-22 16:03:41 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.225: INFO: 	Container webserver ready: true, restart count 0
Jan 22 16:07:43.225: INFO: test-rolling-update-with-lb-58c5c55949-89txc started at 2023-01-22 16:07:23 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.225: INFO: 	Container agnhost ready: true, restart count 0
Jan 22 16:07:43.225: INFO: cilium-dpwnn started at 2023-01-22 15:57:34 +0000 UTC (1+1 container statuses recorded)
Jan 22 16:07:43.225: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Jan 22 16:07:43.225: INFO: 	Container cilium-agent ready: true, restart count 2
Jan 22 16:07:43.225: INFO: bin-false8b2b933f-04ff-47e2-93dd-bb6998081ce8 started at 2023-01-22 16:06:18 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.225: INFO: 	Container bin-false8b2b933f-04ff-47e2-93dd-bb6998081ce8 ready: false, restart count 0
Jan 22 16:07:43.225: INFO: ebs-csi-node-5xvpn started at 2023-01-22 15:57:34 +0000 UTC (0+3 container statuses recorded)
Jan 22 16:07:43.225: INFO: 	Container ebs-plugin ready: true, restart count 0
Jan 22 16:07:43.225: INFO: 	Container liveness-probe ready: true, restart count 0
Jan 22 16:07:43.225: INFO: 	Container node-driver-registrar ready: true, restart count 0
Jan 22 16:07:43.225: INFO: csi-mockplugin-resizer-0 started at 2023-01-22 16:06:47 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.225: INFO: 	Container csi-resizer ready: true, restart count 0
Jan 22 16:07:43.225: INFO: pvc-volume-tester-t7j5v started at 2023-01-22 16:06:54 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.225: INFO: 	Container volume-tester ready: true, restart count 0
Jan 22 16:07:43.225: INFO: coredns-867df8f45c-7m6hf started at 2023-01-22 15:58:04 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.225: INFO: 	Container coredns ready: true, restart count 0
Jan 22 16:07:43.225: INFO: ss-0 started at 2023-01-22 16:01:47 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.225: INFO: 	Container webserver ready: true, restart count 0
Jan 22 16:07:43.225: INFO: netserver-1 started at 2023-01-22 16:03:48 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.225: INFO: 	Container webserver ready: true, restart count 0
Jan 22 16:07:43.350: INFO: 
Latency metrics for node ip-172-20-39-82.us-east-2.compute.internal
Jan 22 16:07:43.351: INFO: 
Logging node info for node ip-172-20-40-246.us-east-2.compute.internal
Jan 22 16:07:43.381: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-40-246.us-east-2.compute.internal    a4283492-f7a9-4ac6-aaab-f667bea07522 12609 0 2023-01-22 15:57:26 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-east-2 failure-domain.beta.kubernetes.io/zone:us-east-2a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-40-246.us-east-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-east-2a topology.hostpath.csi/node:ip-172-20-40-246.us-east-2.compute.internal topology.kubernetes.io/region:us-east-2 topology.kubernetes.io/zone:us-east-2a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-provisioning-2355":"ip-172-20-40-246.us-east-2.compute.internal","csi-hostpath-volume-expand-1757":"ip-172-20-40-246.us-east-2.compute.internal","ebs.csi.aws.com":"i-0a5034e547859f6df"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kubelet Update v1 2023-01-22 15:57:26 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kops-controller Update v1 2023-01-22 15:57:27 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-22 15:57:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {e2e.test Update v1 2023-01-22 16:02:30 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kube-controller-manager Update v1 2023-01-22 16:07:35 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-22 16:07:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///us-east-2a/i-0a5034e547859f6df,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{47441653760 0} {<nil>} 46329740Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4054806528 0} {<nil>} 3959772Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{42697488314 0} {<nil>} 42697488314 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3949948928 0} {<nil>} 3857372Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-22 16:07:39 +0000 UTC,LastTransitionTime:2023-01-22 15:57:26 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-22 16:07:39 +0000 UTC,LastTransitionTime:2023-01-22 15:57:26 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-22 16:07:39 +0000 UTC,LastTransitionTime:2023-01-22 15:57:26 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-22 16:07:39 +0000 UTC,LastTransitionTime:2023-01-22 15:57:47 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.40.246,},NodeAddress{Type:ExternalIP,Address:3.22.242.15,},NodeAddress{Type:Hostname,Address:ip-172-20-40-246.us-east-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-40-246.us-east-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-22-242-15.us-east-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2ac3af6059e59b70f877f01430277f,SystemUUID:ec2ac3af-6059-e59b-70f8-77f01430277f,BootID:f03e4ca2-c137-40c8-ba1b-f79f8d612996,KernelVersion:5.15.86-flatcar,OSImage:Flatcar Container Linux by Kinvolk 3446.1.0 (Oklo),ContainerRuntimeVersion:docker://20.10.21,KubeletVersion:v1.23.16,KubeProxyVersion:v1.23.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456551323,},ContainerImage{Names:[nginx@sha256:b8f2383a95879e1ae064940d9a200f67a6c79e710ed82ac42263397367e7cc4e nginx:latest],SizeBytes:141788385,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126853136,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123774598,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.16 registry.k8s.io/kube-proxy-amd64:v1.23.16],SizeBytes:110829687,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:94469968,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56366111,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56363051,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53940090,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53515772,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53483352,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:b9e2958a3dd879e3cf11142228c6d073d0fc4ea2e857c3be6f4fb0ab5fb2c937 k8s.gcr.io/e2e-test-images/nonroot:1.2],SizeBytes:42320060,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29972082,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19646681,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19575418,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18683218,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17664775,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17605286,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17159270,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16028126,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-provisioning-2355^e12f5e47-9a6e-11ed-8b80-beea856c54d4 kubernetes.io/csi/ebs.csi.aws.com^vol-0bc00114c135a19c9],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-provisioning-2355^e12f5e47-9a6e-11ed-8b80-beea856c54d4,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0bc00114c135a19c9,DevicePath:,},},Config:nil,},}
Jan 22 16:07:43.381: INFO: 
Logging kubelet events for node ip-172-20-40-246.us-east-2.compute.internal
Jan 22 16:07:43.417: INFO: 
Logging pods the kubelet thinks is on node ip-172-20-40-246.us-east-2.compute.internal
Jan 22 16:07:43.492: INFO: ebs-csi-node-vrjbd started at 2023-01-22 15:57:27 +0000 UTC (0+3 container statuses recorded)
Jan 22 16:07:43.492: INFO: 	Container ebs-plugin ready: true, restart count 0
Jan 22 16:07:43.492: INFO: 	Container liveness-probe ready: true, restart count 0
Jan 22 16:07:43.492: INFO: 	Container node-driver-registrar ready: true, restart count 0
Jan 22 16:07:43.492: INFO: host-test-container-pod started at <nil> (0+0 container statuses recorded)
Jan 22 16:07:43.492: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-26ll5 started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.492: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:07:43.492: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-jp7lk started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.492: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:07:43.492: INFO: test-rolling-update-with-lb-58c5c55949-xscb2 started at 2023-01-22 16:07:21 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.492: INFO: 	Container agnhost ready: true, restart count 0
Jan 22 16:07:43.492: INFO: externalname-service-kvtfg started at 2023-01-22 16:07:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.492: INFO: 	Container externalname-service ready: true, restart count 0
Jan 22 16:07:43.492: INFO: ss2-1 started at 2023-01-22 16:07:29 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.492: INFO: 	Container webserver ready: true, restart count 0
Jan 22 16:07:43.492: INFO: netserver-2 started at 2023-01-22 16:03:41 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.492: INFO: 	Container webserver ready: true, restart count 0
Jan 22 16:07:43.492: INFO: cilium-s627s started at 2023-01-22 15:57:27 +0000 UTC (1+1 container statuses recorded)
Jan 22 16:07:43.492: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Jan 22 16:07:43.492: INFO: 	Container cilium-agent ready: true, restart count 2
Jan 22 16:07:43.492: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-2kqq5 started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.492: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:07:43.492: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-2xdb2 started at 2023-01-22 16:02:31 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.492: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:07:43.492: INFO: netserver-2 started at 2023-01-22 16:03:49 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.492: INFO: 	Container webserver ready: true, restart count 0
Jan 22 16:07:43.492: INFO: startup-script started at 2023-01-22 16:07:34 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.492: INFO: 	Container startup-script ready: true, restart count 0
Jan 22 16:07:43.492: INFO: csi-hostpathplugin-0 started at 2023-01-22 16:07:13 +0000 UTC (0+7 container statuses recorded)
Jan 22 16:07:43.492: INFO: 	Container csi-attacher ready: true, restart count 0
Jan 22 16:07:43.492: INFO: 	Container csi-provisioner ready: true, restart count 0
Jan 22 16:07:43.492: INFO: 	Container csi-resizer ready: true, restart count 0
Jan 22 16:07:43.492: INFO: 	Container csi-snapshotter ready: true, restart count 0
Jan 22 16:07:43.492: INFO: 	Container hostpath ready: true, restart count 0
Jan 22 16:07:43.492: INFO: 	Container liveness-probe ready: true, restart count 0
Jan 22 16:07:43.492: INFO: 	Container node-driver-registrar ready: true, restart count 0
Jan 22 16:07:43.492: INFO: test-rolling-update-with-lb-58c5c56bd8-vzmzm started at <nil> (0+0 container statuses recorded)
Jan 22 16:07:43.492: INFO: pod-fd96f517-ffda-47df-ae34-d9de2cf5b974 started at 2023-01-22 16:07:36 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.492: INFO: 	Container write-pod ready: false, restart count 0
Jan 22 16:07:43.492: INFO: csi-hostpathplugin-0 started at 2023-01-22 16:07:28 +0000 UTC (0+7 container statuses recorded)
Jan 22 16:07:43.492: INFO: 	Container csi-attacher ready: true, restart count 0
Jan 22 16:07:43.492: INFO: 	Container csi-provisioner ready: true, restart count 0
Jan 22 16:07:43.492: INFO: 	Container csi-resizer ready: true, restart count 0
Jan 22 16:07:43.492: INFO: 	Container csi-snapshotter ready: true, restart count 0
Jan 22 16:07:43.492: INFO: 	Container hostpath ready: true, restart count 0
Jan 22 16:07:43.492: INFO: 	Container liveness-probe ready: true, restart count 0
Jan 22 16:07:43.492: INFO: 	Container node-driver-registrar ready: true, restart count 0
Jan 22 16:07:43.492: INFO: hostexec-ip-172-20-40-246.us-east-2.compute.internal-lsfwr started at 2023-01-22 16:07:28 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.492: INFO: 	Container agnhost-container ready: true, restart count 0
Jan 22 16:07:43.492: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-pdmh6 started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.492: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:07:43.492: INFO: pod-subpath-test-dynamicpv-cvms started at <nil> (0+0 container statuses recorded)
Jan 22 16:07:43.492: INFO: host-test-container-pod started at 2023-01-22 16:07:32 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.492: INFO: 	Container agnhost-container ready: false, restart count 0
Jan 22 16:07:43.492: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-lrc95 started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.492: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:07:43.492: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-bgclt started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.492: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:07:43.492: INFO: ss-2 started at 2023-01-22 16:07:33 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.492: INFO: 	Container webserver ready: false, restart count 0
Jan 22 16:07:43.492: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-jp49t started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.492: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:07:43.492: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-tg6vg started at 2023-01-22 16:02:31 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.492: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:07:43.692: INFO: 
Latency metrics for node ip-172-20-40-246.us-east-2.compute.internal
Jan 22 16:07:43.693: INFO: 
Logging node info for node ip-172-20-45-186.us-east-2.compute.internal
Jan 22 16:07:43.722: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-45-186.us-east-2.compute.internal    d0ea7688-ae6f-4b0b-b4c0-87950ac02237 9896 0 2023-01-22 15:57:24 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-east-2 failure-domain.beta.kubernetes.io/zone:us-east-2a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-45-186.us-east-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:us-east-2a topology.hostpath.csi/node:ip-172-20-45-186.us-east-2.compute.internal topology.kubernetes.io/region:us-east-2 topology.kubernetes.io/zone:us-east-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0ce264dea7066c689"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kops-controller Update v1 2023-01-22 15:57:24 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-01-22 15:57:24 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2023-01-22 15:57:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {e2e.test Update v1 2023-01-22 16:02:30 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kubelet Update v1 2023-01-22 16:06:16 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///us-east-2a/i-0ce264dea7066c689,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{47441653760 0} {<nil>} 46329740Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4054806528 0} {<nil>} 3959772Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{42697488314 0} {<nil>} 42697488314 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3949948928 0} {<nil>} 3857372Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-22 16:06:16 +0000 UTC,LastTransitionTime:2023-01-22 15:57:24 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-22 16:06:16 +0000 UTC,LastTransitionTime:2023-01-22 15:57:24 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-22 16:06:16 +0000 UTC,LastTransitionTime:2023-01-22 15:57:24 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-22 16:06:16 +0000 UTC,LastTransitionTime:2023-01-22 15:57:55 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.45.186,},NodeAddress{Type:ExternalIP,Address:18.118.166.212,},NodeAddress{Type:Hostname,Address:ip-172-20-45-186.us-east-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-45-186.us-east-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-118-166-212.us-east-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2308cb46db6b634f318d31f03adcbc,SystemUUID:ec2308cb-46db-6b63-4f31-8d31f03adcbc,BootID:c1380477-0aee-480e-8dd1-20b7d9813f9d,KernelVersion:5.15.86-flatcar,OSImage:Flatcar Container Linux by Kinvolk 3446.1.0 (Oklo),ContainerRuntimeVersion:docker://20.10.21,KubeletVersion:v1.23.16,KubeProxyVersion:v1.23.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456551323,},ContainerImage{Names:[nginx@sha256:b8f2383a95879e1ae064940d9a200f67a6c79e710ed82ac42263397367e7cc4e nginx:latest],SizeBytes:141788385,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:126853136,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:123774598,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.16 registry.k8s.io/kube-proxy-amd64:v1.23.16],SizeBytes:110829687,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:94469968,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:56366111,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:56363051,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:53940090,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:53515772,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:53484608,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:53483352,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:49160266,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:29972082,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:19646681,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19575418,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:18683218,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:17664775,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17605286,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:17159270,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16028126,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:4202236,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db k8s.gcr.io/pause:3.6 registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Jan 22 16:07:43.723: INFO: 
Logging kubelet events for node ip-172-20-45-186.us-east-2.compute.internal
Jan 22 16:07:43.755: INFO: 
Logging pods the kubelet thinks is on node ip-172-20-45-186.us-east-2.compute.internal
Jan 22 16:07:43.790: INFO: hostexec-ip-172-20-45-186.us-east-2.compute.internal-v47t5 started at 2023-01-22 16:07:24 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.790: INFO: 	Container agnhost-container ready: true, restart count 0
Jan 22 16:07:43.790: INFO: netserver-3 started at 2023-01-22 16:03:41 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.790: INFO: 	Container webserver ready: true, restart count 0
Jan 22 16:07:43.790: INFO: cilium-zt6nc started at 2023-01-22 15:57:25 +0000 UTC (1+1 container statuses recorded)
Jan 22 16:07:43.790: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Jan 22 16:07:43.790: INFO: 	Container cilium-agent ready: true, restart count 2
Jan 22 16:07:43.790: INFO: local-injector started at <nil> (0+0 container statuses recorded)
Jan 22 16:07:43.790: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-kxcmp started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.790: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:07:43.790: INFO: boom-server started at 2023-01-22 16:07:25 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.790: INFO: 	Container boom-server ready: true, restart count 0
Jan 22 16:07:43.790: INFO: test-container-pod started at <nil> (0+0 container statuses recorded)
Jan 22 16:07:43.790: INFO: client-containers-5b54d3a3-e625-4873-b73b-4b28855a9d12 started at 2023-01-22 16:07:23 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.790: INFO: 	Container agnhost-container ready: true, restart count 0
Jan 22 16:07:43.790: INFO: test-container-pod started at 2023-01-22 16:07:32 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.790: INFO: 	Container webserver ready: true, restart count 0
Jan 22 16:07:43.790: INFO: netserver-3 started at 2023-01-22 16:03:49 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.790: INFO: 	Container webserver ready: true, restart count 0
Jan 22 16:07:43.790: INFO: cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6-qjt7c started at 2023-01-22 16:02:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.790: INFO: 	Container cleanup40-9d29053b-f42b-480d-a040-19ad8db9d9f6 ready: true, restart count 0
Jan 22 16:07:43.790: INFO: ebs-csi-node-45fpb started at 2023-01-22 15:57:25 +0000 UTC (0+3 container statuses recorded)
Jan 22 16:07:43.790: INFO: 	Container ebs-plugin ready: true, restart count 0
Jan 22 16:07:43.790: INFO: 	Container liveness-probe ready: true, restart count 0
Jan 22 16:07:43.790: INFO: 	Container node-driver-registrar ready: true, restart count 0
Jan 22 16:07:43.790: INFO: ss2-0 started at 2023-01-22 16:07:25 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.790: INFO: 	Container webserver ready: true, restart count 0
Jan 22 16:07:43.790: INFO: externalname-service-4696r started at 2023-01-22 16:07:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.791: INFO: 	Container externalname-service ready: true, restart count 0
Jan 22 16:07:43.791: INFO: var-expansion-11685963-07cf-4f4c-b8ee-eaf84f6c4f7f started at 2023-01-22 16:07:30 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:43.791: INFO: 	Container dapi-container ready: false, restart count 0
Jan 22 16:07:43.920: INFO: 
Latency metrics for node ip-172-20-45-186.us-east-2.compute.internal
Jan 22 16:07:43.920: INFO: 
Logging node info for node ip-172-20-52-12.us-east-2.compute.internal
Jan 22 16:07:43.950: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-52-12.us-east-2.compute.internal    33149587-e10e-48d5-abaf-ca8103e85c14 6683 0 2023-01-22 15:56:13 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-east-2 failure-domain.beta.kubernetes.io/zone:us-east-2a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-52-12.us-east-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/control-plane: node-role.kubernetes.io/master: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-east-2a topology.kubernetes.io/region:us-east-2 topology.kubernetes.io/zone:us-east-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0f06483805887653c"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kubelet Update v1 2023-01-22 15:56:13 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {protokube Update v1 2023-01-22 15:56:32 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kops-controller Update v1 2023-01-22 15:56:36 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/master":{}}}} } {kube-controller-manager Update v1 2023-01-22 15:56:54 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-01-22 15:57:53 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///us-east-2a/i-0f06483805887653c,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{47441653760 0} {<nil>} 46329740Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3895427072 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{42697488314 0} {<nil>} 42697488314 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3790569472 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-22 16:03:11 +0000 UTC,LastTransitionTime:2023-01-22 15:56:13 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-22 16:03:11 +0000 UTC,LastTransitionTime:2023-01-22 15:56:13 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-22 16:03:11 +0000 UTC,LastTransitionTime:2023-01-22 15:56:13 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-22 16:03:11 +0000 UTC,LastTransitionTime:2023-01-22 15:56:54 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.52.12,},NodeAddress{Type:ExternalIP,Address:3.16.137.247,},NodeAddress{Type:Hostname,Address:ip-172-20-52-12.us-east-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-52-12.us-east-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-16-137-247.us-east-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec221e40526c51c23560b6cf49b17a65,SystemUUID:ec221e40-526c-51c2-3560-b6cf49b17a65,BootID:89f5d760-653a-4491-8f94-5adaf8195cda,KernelVersion:5.15.86-flatcar,OSImage:Flatcar Container Linux by Kinvolk 3446.1.0 (Oklo),ContainerRuntimeVersion:docker://20.10.21,KubeletVersion:v1.23.16,KubeProxyVersion:v1.23.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:66a453db625abb268f4b3bbefc5a34a171d81e6e8796cecca54cfd71775c77c4 registry.k8s.io/etcdadm/etcd-manager:v3.0.20221209],SizeBytes:662196755,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:456551323,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64:v1.23.16 registry.k8s.io/kube-apiserver-amd64:v1.23.16],SizeBytes:129993155,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64:v1.23.16 registry.k8s.io/kube-controller-manager-amd64:v1.23.16],SizeBytes:119933673,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.23.16 registry.k8s.io/kube-proxy-amd64:v1.23.16],SizeBytes:110829687,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:110016956,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:105636625,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:102694452,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:94469968,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:57729376,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:55458084,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:54841901,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64:v1.23.16 registry.k8s.io/kube-scheduler-amd64:v1.23.16],SizeBytes:51845852,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:19575418,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:17605286,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:8785817,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:682696,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Jan 22 16:07:43.951: INFO: 
Logging kubelet events for node ip-172-20-52-12.us-east-2.compute.internal
Jan 22 16:07:43.983: INFO: 
Logging pods the kubelet thinks is on node ip-172-20-52-12.us-east-2.compute.internal
Jan 22 16:07:44.021: INFO: etcd-manager-events-ip-172-20-52-12.us-east-2.compute.internal started at 2023-01-22 15:55:26 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:44.022: INFO: 	Container etcd-manager ready: true, restart count 0
Jan 22 16:07:44.022: INFO: kube-apiserver-ip-172-20-52-12.us-east-2.compute.internal started at 2023-01-22 15:55:26 +0000 UTC (0+2 container statuses recorded)
Jan 22 16:07:44.022: INFO: 	Container healthcheck ready: true, restart count 0
Jan 22 16:07:44.022: INFO: 	Container kube-apiserver ready: true, restart count 1
Jan 22 16:07:44.022: INFO: kube-controller-manager-ip-172-20-52-12.us-east-2.compute.internal started at 2023-01-22 15:55:26 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:44.022: INFO: 	Container kube-controller-manager ready: true, restart count 2
Jan 22 16:07:44.022: INFO: kube-scheduler-ip-172-20-52-12.us-east-2.compute.internal started at 2023-01-22 15:55:26 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:44.022: INFO: 	Container kube-scheduler ready: true, restart count 0
Jan 22 16:07:44.022: INFO: cilium-tqjcp started at 2023-01-22 15:56:35 +0000 UTC (1+1 container statuses recorded)
Jan 22 16:07:44.022: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Jan 22 16:07:44.022: INFO: 	Container cilium-agent ready: true, restart count 0
Jan 22 16:07:44.022: INFO: ebs-csi-controller-54b65c69b6-ljh8m started at 2023-01-22 15:56:35 +0000 UTC (0+5 container statuses recorded)
Jan 22 16:07:44.022: INFO: 	Container csi-attacher ready: true, restart count 0
Jan 22 16:07:44.022: INFO: 	Container csi-provisioner ready: true, restart count 0
Jan 22 16:07:44.022: INFO: 	Container csi-resizer ready: true, restart count 0
Jan 22 16:07:44.022: INFO: 	Container ebs-plugin ready: true, restart count 0
Jan 22 16:07:44.022: INFO: 	Container liveness-probe ready: true, restart count 0
Jan 22 16:07:44.022: INFO: dns-controller-7db8b74b99-5rkk4 started at 2023-01-22 15:56:35 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:44.022: INFO: 	Container dns-controller ready: true, restart count 0
Jan 22 16:07:44.022: INFO: etcd-manager-cilium-ip-172-20-52-12.us-east-2.compute.internal started at 2023-01-22 15:55:26 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:44.022: INFO: 	Container etcd-manager ready: true, restart count 0
Jan 22 16:07:44.022: INFO: etcd-manager-main-ip-172-20-52-12.us-east-2.compute.internal started at 2023-01-22 15:55:26 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:44.022: INFO: 	Container etcd-manager ready: true, restart count 0
Jan 22 16:07:44.022: INFO: kops-controller-c9664 started at 2023-01-22 15:56:35 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:44.022: INFO: 	Container kops-controller ready: true, restart count 0
Jan 22 16:07:44.022: INFO: ebs-csi-node-288bv started at 2023-01-22 15:56:35 +0000 UTC (0+3 container statuses recorded)
Jan 22 16:07:44.022: INFO: 	Container ebs-plugin ready: true, restart count 0
Jan 22 16:07:44.022: INFO: 	Container liveness-probe ready: true, restart count 0
Jan 22 16:07:44.022: INFO: 	Container node-driver-registrar ready: true, restart count 0
Jan 22 16:07:44.022: INFO: cilium-operator-68bc649ff6-vvr4s started at 2023-01-22 15:56:35 +0000 UTC (0+1 container statuses recorded)
Jan 22 16:07:44.022: INFO: 	Container cilium-operator ready: true, restart count 0
Jan 22 16:07:44.182: INFO: 
Latency metrics for node ip-172-20-52-12.us-east-2.compute.internal
Jan 22 16:07:44.182: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
�[1mSTEP�[0m: Destroying namespace "provisioning-7201" for this suite.

				Click to see stdout/stderrfrom junit_11.xml

Find pod-subpath-test-inlinevolume-r9xz mentions in log files

kubetest2 Test 23m19s

exit status 255
				from junit_runner.xml

Filter through log files

Show 779 Passed Tests

Show 6528 Skipped Tests

Result	FAILURE
Tests	4 failed / 779 succeeded
Started	2023-01-22 15:52
Elapsed	33m25s
Revision	master