-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
Jenkins 2.263.2
Kuberenetes Plugin 1.28.4
Jenkins helm chart deployed to AWS EKS K8s worker nodes.
Chart version is 3.1.2.
Jenkins master and slave used to work until I needed to re-deploy Jenkins pod after the underlying EC2 needed to be restarted to fix vulnerabilities of linux packages.
Followed the kubernetes plugin doc to setup Cloud config: https://github.com/jenkinsci/kubernetes-plugin
As in the screenshot, connection to Jenkins is successful using "Test Connection" button as Jenkins pod is within AWS EKS cluster.
When I trigger Jenkins job, slave pod terminates.
Here are logs:
$ k logs -n jenkins -c jnlp -f xxx-master-25-z0h57-2hfpd-7632l
Mar 18, 2021 8:29:30 PM hudson.remoting.jnlp.Main createEngine INFO: Setting up agent: xxx-master-25-z0h57-2hfpd-7632l Mar 18, 2021 8:29:30 PM hudson.remoting.jnlp.Main$CuiListener <init> INFO: Jenkins agent is running in headless mode. Mar 18, 2021 8:29:30 PM hudson.remoting.Engine startEngine INFO: Using Remoting version: 4.3 Mar 18, 2021 8:29:30 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir INFO: Using /home/jenkins/agent/remoting as a remoting work directory Mar 18, 2021 8:29:30 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging INFO: Both error and output logs will be printed to /home/jenkins/agent/remoting Mar 18, 2021 8:29:30 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Locating server among [http://jenkins:8080/] Mar 18, 2021 8:29:30 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping] Mar 18, 2021 8:29:35 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver isPortVisible WARNING: connect timed out Mar 18, 2021 8:29:35 PM hudson.remoting.jnlp.Main$CuiListener error SEVERE: http://jenkins:8080/ provided port:50000 is not reachable java.io.IOException: http://jenkins:8080/ provided port:50000 is not reachable at org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver.resolve(JnlpAgentEndpointResolver.java:314) at hudson.remoting.Engine.innerRun(Engine.java:693) at hudson.remoting.Engine.run(Engine.java:518)
Verified the endpoint `/tcpSlaveAgentListener` from a curl pod in jenkins namespace
k apply -f ../../tests/pod_curl.yaml
k exec -it curl -n jenkins sh / $ curl jenkins:8080/tcpSlaveAgentListener/ -v * Trying 172.20.35.230:8080... * Connected to jenkins (172.20.35.230) port 8080 (#0) > GET /tcpSlaveAgentListener/ HTTP/1.1 > Host: jenkins:8080 > User-Agent: curl/7.75.0-DEV > Accept: */* > * Mark bundle as not supporting multiuse < HTTP/1.1 200 OK # <----- works! < Date: Thu, 18 Mar 2021 19:49:34 GMT < X-Content-Type-Options: nosniff < Content-Type: text/plain;charset=utf-8 < X-Hudson-JNLP-Port: 50000 < X-Jenkins-JNLP-Port: 50000 < X-Instance-Identity: MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAplLpc8tR8VSYXA9MFqeJT7UQl8RjGhN9rnbhZJiK+RRkDIs9IsOX0vsdP6WuZkUHr49DxZYpuZOJcTDYoctzTr+jOS5JB7pGE6zpJI7YsrcS0f5S/Umlssdj5vYf6D3oHj1X/afrchvhWCJRRG94JIjxYjN0Cac5P8whd8Q2QoNPEncTY9MfDet8yn1PxXd0uq2LH8LbwOsDszsWOpxw2ACekpniauCWyw20B1WiAoj9l4DplyugvWCZQqCzl9ls0N7xe7FXZctMxP3IBZhh/zhoUbcS8y4tNP6fLNkLAVWMFyqYa6GVww7RpyGgnll9RCvQTR2K+cXzWBITop29pwIDAQAB < X-Jenkins-Agent-Protocols: JNLP4-connect, Ping < X-Remoting-Minimum-Version: 3.14 < Content-Length: 12 < Server: Jetty(9.4.33.v20201020) < Jenkins * Connection #0 to host jenkins left intact
However, the private endpoint (with AWS VPN) `/tcpSlaveAgentListener` used to work but it doesn't now, not sure if this is related to the error "provided port:50000 is not reachable"
# used to work $ curl http://internal-xxxx-xxxx.us-east-1.elb.amazonaws.com/tcpSlaveAgentListener/ -v * Trying 10.1.xx.xx... * TCP_NODELAY set * Connected to internal-xxxx-xxxx.us-east-1.elb.amazonaws.com (10.1.xx.xx) port 80 (#0) > GET /tcpSlaveAgentListener/ HTTP/1.1 > Host: internal-xxxx-xxxx.us-east-1.elb.amazonaws.com > User-Agent: curl/7.54.0 > Accept: */* > < HTTP/1.1 200 OK < date: Fri, 12 Jun 2020 11:50:37 GMT < x-content-type-options: nosniff < content-type: text/plain;charset=utf-8 < x-hudson-jnlp-port: 50000 < x-jenkins-jnlp-port: 50000 < x-instance-identity: MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAuSNmwO+JEpFTaJvuIb5o8+gr311aFqAfRV8Hh97mJHZmGBqG7kGJf74tc6hr5cREVRD+vw8giqaUzyvALu4GomUVJFpo0PzCXaRjphRIjkdhis7oZ8utdtCl9CdNGr9yXVZq4hp+znCm3Rg9XNlJ1u8pWLGihk4vz+2phkXBQ0rOCk203L8KuQ8CeEgbSvSQHwtyiSUixAVO1AVZ0uWBNqBdzwKu6GuaAqAU1lUErJrxKk+NVqZJ5KiOAMnbVbsEwAou3ySIBZPeSsALsez/y2BKJfJD8gdvqRmVp6GNsYXU56IbsM9s8WyAmVwP85h52Svl8sSr3UsbNEOcZsy5VwIDAQAB < x-jenkins-agent-protocols: JNLP4-connect, Ping < x-remoting-minimum-version: 3.14 < content-length: 12 < server: istio-envoy < x-envoy-upstream-service-time: 2 < Jenkins # right now doesn't work curl http://internal-xxxx-xxxx.us-east-1.elb.amazonaws.com/tcpSlaveAgentListener/ -v * Trying 10.1.xx.xx... * TCP_NODELAY set * Connected to internal-xxxx-xxxx.us-east-1.elb.amazonaws.com (10.1.xx.xx) port 80 (#0) > GET /tcpSlaveAgentListener/ HTTP/1.1 > Host: internal-xxxx-xxxx.us-east-1.elb.amazonaws.com > User-Agent: curl/7.54.0 > Accept: */* > < HTTP/1.1 404 Not Found < date: Thu, 18 Mar 2021 20:50:58 GMT < server: istio-envoy < Content-Length: 0 < Connection: keep-alive < * Connection #0 to host internal-xxx-xxxx.us-east-1.elb.amazonaws.com left intact
I've tried setting JENKINS_URL=http://jenkins:8080, to no avail.
When I set JENKINS_TUNNEL=jenkins:50000, then jenkins slave pod hangs
$ k logs -n jenkins -c jnlp -f xxx-master-24-ltvqp-48lxv-q122c
Mar 18, 2021 8:28:40 PM hudson.remoting.jnlp.Main createEngine INFO: Setting up agent: xxxx-24-ltvqp-48lxv-q122c Mar 18, 2021 8:28:40 PM hudson.remoting.jnlp.Main$CuiListener <init> INFO: Jenkins agent is running in headless mode. Mar 18, 2021 8:28:40 PM hudson.remoting.Engine startEngine INFO: Using Remoting version: 4.3 Mar 18, 2021 8:28:40 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir INFO: Using /home/jenkins/agent/remoting as a remoting work directory Mar 18, 2021 8:28:40 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging INFO: Both error and output logs will be printed to /home/jenkins/agent/remoting Mar 18, 2021 8:28:40 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Locating server among [http://jenkins:8080/] Mar 18, 2021 8:28:40 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping] Mar 18, 2021 8:28:40 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve INFO: Remoting TCP connection tunneling is enabled. Skipping the TCP Agent Listener Port availability check Mar 18, 2021 8:28:40 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Agent discovery successful Agent address: jenkins Agent port: 50000 Identity: fc:7f:01:98:49:4a:b5:ac:51:bd:73:6c:f7:b3:08:71 Mar 18, 2021 8:28:40 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Handshaking Mar 18, 2021 8:28:40 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Connecting to jenkins:50000 # <------ hangs here for 2 mins and eventually pod terminates
I've looked through and tried these:
- https://stackoverflow.com/questions/44180595/tcpslaveagentlistener-not-found-on-jenkins-server
- https://stackoverflow.com/questions/58719522/tcpslaveagentlistener-is-invalid-404-not-found - https://github.com/jenkinsci/docker/issues/788
- https://programmer.ink/think/installing-jenkins-on-k8s-and-common-problems.html
- https://issues.jenkins.io/browse/JENKINS-63832
