Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-67269

Kubernetes workload fails after few hours

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Critical Critical
    • kubernetes-plugin
    • None
    • - 3 node Rancher 2.6.2 cluster on OpenSUSe 15.3 Leap, kernel 5.3.18-59.34-default for workloads
      - single node k3s cluster Rancher 2.6.2 for management
      - Jenkins 2.303.3, Java 11
      - Kubernetes plugin 1.30.11

      After about six hours, workload fails with pods in state "not ready" with following logs in Jenkins, until failure it runs repetedly without issues:

      java.net.ProtocolException: Expected HTTP 101 response but was '502 Bad Gateway'

	at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229)

	at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196)

	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203)

	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)

	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

	at java.base/java.lang.Thread.run(Thread.java:829)

ERROR: Process exited immediately after creation. See output below

Executing sh script inside container jnks-fio-inside-mount-container of pod patamat-k8s-nfsv4-1-fio-inside-mount-opensuse-579-dnlbw-f-hrf8t



Process exited immediately after creation. Check logs above for more details.

      and following logs in rancher on jnlp container:

      Warning: SECRET is defined twice in command-line arguments and the environment variable
Warning: AGENT_NAME is defined twice in command-line arguments and the environment variable
Dec 01, 2021 2:52:05 PM hudson.remoting.jnlp.Main createEngine
INFO: Setting up agent: server-k8s-nfsv4-1-fio-inside-mount-opensuse-506-kdbmg-7h6-g27np
Dec 01, 2021 2:52:05 PM hudson.remoting.jnlp.Main$CuiListener <init>
INFO: Jenkins agent is running in headless mode.
Dec 01, 2021 2:52:05 PM hudson.remoting.Engine startEngine
INFO: Using Remoting version: 4.11
Dec 01, 2021 2:52:05 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using /home/jenkins/agent/remoting as a remoting work directory
Dec 01, 2021 2:52:05 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
INFO: Both error and output logs will be printed to /home/jenkins/agent/remoting
Dec 01, 2021 2:52:05 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: WebSocket connection open
Dec 01, 2021 2:52:05 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connected
Dec 01, 2021 2:56:16 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave$SlaveDisconnector call
INFO: Disabled agent engine reconnects.
Dec 01, 2021 2:56:20 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Write side closed
Dec 01, 2021 2:56:20 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Read side closed
Dec 01, 2021 2:56:20 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Terminated
Dec 01, 2021 2:56:20 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Read side closed

            Unassigned Unassigned
            bganicky bga
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: