Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-65176

K8s Jenkins slave pod error "SEVERE: http://jenkins:8080/ provided port:50000 is not reachable"

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: kubernetes-plugin
    • Labels:
      None
    • Environment:
      Jenkins 2.263.2
      Kuberenetes Plugin 1.28.4
    • Similar Issues:

      Description

       

      Jenkins helm chart deployed to AWS EKS K8s worker nodes.

      Chart version is 3.1.2.

       

      Jenkins master and slave used to work until I needed to re-deploy Jenkins pod after the underlying EC2 needed to be restarted to fix vulnerabilities of linux packages.

      Followed the kubernetes plugin doc to setup Cloud config: https://github.com/jenkinsci/kubernetes-plugin

       

      As in the screenshot, connection to Jenkins is successful using "Test Connection" button as Jenkins pod is within AWS EKS cluster.

       

      When I trigger Jenkins job, slave pod terminates.

       

      Here are logs:

       

      $ k logs -n jenkins -c jnlp -f xxx-master-25-z0h57-2hfpd-7632l 
      Mar 18, 2021 8:29:30 PM hudson.remoting.jnlp.Main createEngine
      INFO: Setting up agent: xxx-master-25-z0h57-2hfpd-7632l
      Mar 18, 2021 8:29:30 PM hudson.remoting.jnlp.Main$CuiListener <init>
      INFO: Jenkins agent is running in headless mode.
      Mar 18, 2021 8:29:30 PM hudson.remoting.Engine startEngine
      INFO: Using Remoting version: 4.3
      Mar 18, 2021 8:29:30 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
      INFO: Using /home/jenkins/agent/remoting as a remoting work directory
      Mar 18, 2021 8:29:30 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
      INFO: Both error and output logs will be printed to /home/jenkins/agent/remoting
      Mar 18, 2021 8:29:30 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Locating server among [http://jenkins:8080/]
      Mar 18, 2021 8:29:30 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
      INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
      Mar 18, 2021 8:29:35 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver isPortVisible
      WARNING: connect timed out
      Mar 18, 2021 8:29:35 PM hudson.remoting.jnlp.Main$CuiListener error
      SEVERE: http://jenkins:8080/ provided port:50000 is not reachable
      java.io.IOException: http://jenkins:8080/ provided port:50000 is not reachable
       at org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver.resolve(JnlpAgentEndpointResolver.java:314)
       at hudson.remoting.Engine.innerRun(Engine.java:693)
       at hudson.remoting.Engine.run(Engine.java:518)

       

       

       
      Verified the endpoint `/tcpSlaveAgentListener` from a curl pod in jenkins namespace
       

      k apply -f ../../tests/pod_curl.yaml 
       
      k exec -it curl -n jenkins sh 
       / $ curl jenkins:8080/tcpSlaveAgentListener/ -v *   Trying 172.20.35.230:8080... * Connected to jenkins (172.20.35.230) port 8080 (#0) > GET /tcpSlaveAgentListener/ HTTP/1.1 > Host: jenkins:8080 > User-Agent: curl/7.75.0-DEV > Accept: */* >  * Mark bundle as not supporting multiuse < HTTP/1.1 200 OK   # <----- works! < Date: Thu, 18 Mar 2021 19:49:34 GMT < X-Content-Type-Options: nosniff < Content-Type: text/plain;charset=utf-8 < X-Hudson-JNLP-Port: 50000 < X-Jenkins-JNLP-Port: 50000 < X-Instance-Identity: MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAplLpc8tR8VSYXA9MFqeJT7UQl8RjGhN9rnbhZJiK+RRkDIs9IsOX0vsdP6WuZkUHr49DxZYpuZOJcTDYoctzTr+jOS5JB7pGE6zpJI7YsrcS0f5S/Umlssdj5vYf6D3oHj1X/afrchvhWCJRRG94JIjxYjN0Cac5P8whd8Q2QoNPEncTY9MfDet8yn1PxXd0uq2LH8LbwOsDszsWOpxw2ACekpniauCWyw20B1WiAoj9l4DplyugvWCZQqCzl9ls0N7xe7FXZctMxP3IBZhh/zhoUbcS8y4tNP6fLNkLAVWMFyqYa6GVww7RpyGgnll9RCvQTR2K+cXzWBITop29pwIDAQAB < X-Jenkins-Agent-Protocols: JNLP4-connect, Ping < X-Remoting-Minimum-Version: 3.14 < Content-Length: 12 < Server: Jetty(9.4.33.v20201020) <  
      
         Jenkins * Connection #0 to host jenkins left intact

       

      However, the private endpoint (with AWS VPN) `/tcpSlaveAgentListener` used to work but it doesn't now, not sure if this is related to the error "provided port:50000 is not reachable"
       

      # used to work
      $ curl http://internal-xxxx-xxxx.us-east-1.elb.amazonaws.com/tcpSlaveAgentListener/ -v 
       *   Trying 10.1.xx.xx... * TCP_NODELAY set * Connected to internal-xxxx-xxxx.us-east-1.elb.amazonaws.com (10.1.xx.xx) port 80 (#0) > GET /tcpSlaveAgentListener/ HTTP/1.1 > Host: internal-xxxx-xxxx.us-east-1.elb.amazonaws.com > User-Agent: curl/7.54.0 > Accept: */* >  < HTTP/1.1 200 OK < date: Fri, 12 Jun 2020 11:50:37 GMT < x-content-type-options: nosniff < content-type: text/plain;charset=utf-8 < x-hudson-jnlp-port: 50000 < x-jenkins-jnlp-port: 50000 < x-instance-identity: MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAuSNmwO+JEpFTaJvuIb5o8+gr311aFqAfRV8Hh97mJHZmGBqG7kGJf74tc6hr5cREVRD+vw8giqaUzyvALu4GomUVJFpo0PzCXaRjphRIjkdhis7oZ8utdtCl9CdNGr9yXVZq4hp+znCm3Rg9XNlJ1u8pWLGihk4vz+2phkXBQ0rOCk203L8KuQ8CeEgbSvSQHwtyiSUixAVO1AVZ0uWBNqBdzwKu6GuaAqAU1lUErJrxKk+NVqZJ5KiOAMnbVbsEwAou3ySIBZPeSsALsez/y2BKJfJD8gdvqRmVp6GNsYXU56IbsM9s8WyAmVwP85h52Svl8sSr3UsbNEOcZsy5VwIDAQAB < x-jenkins-agent-protocols: JNLP4-connect, Ping < x-remoting-minimum-version: 3.14 < content-length: 12 < server: istio-envoy < x-envoy-upstream-service-time: 2 <  
         Jenkins
      
      
      # right now doesn't work
      curl http://internal-xxxx-xxxx.us-east-1.elb.amazonaws.com/tcpSlaveAgentListener/ -v
      *   Trying 10.1.xx.xx...
      * TCP_NODELAY set
      * Connected to internal-xxxx-xxxx.us-east-1.elb.amazonaws.com (10.1.xx.xx) port 80 (#0)
      > GET /tcpSlaveAgentListener/ HTTP/1.1
      > Host: internal-xxxx-xxxx.us-east-1.elb.amazonaws.com
      > User-Agent: curl/7.54.0
      > Accept: */*
      > 
      < HTTP/1.1 404 Not Found
      < date: Thu, 18 Mar 2021 20:50:58 GMT
      < server: istio-envoy
      < Content-Length: 0
      < Connection: keep-alive
      < 
      * Connection #0 to host internal-xxx-xxxx.us-east-1.elb.amazonaws.com left intact

       
       

       

      I've tried setting JENKINS_URL=http://jenkins:8080, to no avail.

       

      When I set JENKINS_TUNNEL=jenkins:50000, then jenkins slave pod hangs 

      $ k logs -n jenkins -c jnlp -f xxx-master-24-ltvqp-48lxv-q122c 
      Mar 18, 2021 8:28:40 PM hudson.remoting.jnlp.Main createEngine
      INFO: Setting up agent: xxxx-24-ltvqp-48lxv-q122c
      Mar 18, 2021 8:28:40 PM hudson.remoting.jnlp.Main$CuiListener <init>
      INFO: Jenkins agent is running in headless mode.
      Mar 18, 2021 8:28:40 PM hudson.remoting.Engine startEngine
      INFO: Using Remoting version: 4.3
      Mar 18, 2021 8:28:40 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
      INFO: Using /home/jenkins/agent/remoting as a remoting work directory
      Mar 18, 2021 8:28:40 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
      INFO: Both error and output logs will be printed to /home/jenkins/agent/remoting
      Mar 18, 2021 8:28:40 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Locating server among [http://jenkins:8080/]
      Mar 18, 2021 8:28:40 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
      INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
      Mar 18, 2021 8:28:40 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
      INFO: Remoting TCP connection tunneling is enabled. Skipping the TCP Agent Listener Port availability check
      Mar 18, 2021 8:28:40 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Agent discovery successful
       Agent address: jenkins
       Agent port: 50000
       Identity: fc:7f:01:98:49:4a:b5:ac:51:bd:73:6c:f7:b3:08:71
      Mar 18, 2021 8:28:40 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Handshaking
      Mar 18, 2021 8:28:40 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Connecting to jenkins:50000 # <------ hangs here for 2 mins and eventually pod terminates

      I've looked through and tried these:

        Attachments

          Activity

          There are no comments yet on this issue.

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            hasakura Art
            Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated: