• Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Critical Critical
    • core, remoting
    • None

      We has a serious issue in our Jenkins environment. Some times (or better after a a while) we can not connect any jar agents via command line from agent to built.in node.

       

      We use port 50000 for JNLP communication and the log on agent side throws something like this

       

       

      My first thing was, that we runs behind proxys, firewall ... and something happens there.

      BUT:

      + This is only for new connections. Old (currently connected) connections are still fine.

      + and it  when I execute curl directly in the image it looks to fail too.

       sudo docker exec -it etm_test_jenkins  curl https://localhost:5000/
      curl: (7) Failed to connect to localhost port 5000: Connection refused

      and the connection to Jenkins URL looks fine:
      udo docker exec -it etm_test_jenkins  curl https://localhost:8443/
      curl: (60) SSL certificate problem: self signed certificate in certificate chain
      More details here: https://curl.se/docs/sslcerts.html

       

      Can somebody look here. It is more or less a critical point for us. Sometimes we runs 1 month without this issue, But sometimes (like now) we get the same issue in few days (currently 4 days)

      And one important thing. Restart of Jenkins helps always!!! But this is not applicable solution.

        1. jenkins-agent-connect-error.log
          10 kB
          Andreya Kostov
        2. image-2022-11-27-14-07-09-322.png
          5 kB
          Martin Pokorny
        3. image-2022-11-27-11-01-11-583.png
          124 kB
          Martin Pokorny

          [JENKINS-70161] Blocked JNLP port

          in case it helps

          Martin Pokorny added a comment - in case it helps

          Mark Waite added a comment -

          mpokornyetm you're much more likely to receive help on the Jenkins community forum https://community.jenkins.io or the Jenkins user mailing list than by submitting an issue to the Jenkins issue tracker. This looks like a configuration issue or possibly an intermittent issue with your reverse proxy. More people read those locations that read bug reports submitted to Jenkins core.

          Mark Waite added a comment - mpokornyetm  you're much more likely to receive help on the Jenkins community forum https://community.jenkins.io or the Jenkins user mailing list than by submitting an issue to the Jenkins issue tracker. This looks like a configuration issue or possibly an intermittent issue with your reverse proxy. More people read those locations that read bug reports submitted to Jenkins core.

          Hi markewaite  That was also my first think, Our proxy is broken or something like that. But now, after restart the Jenkins (docker image) it works as well. Also my curl command works. Therefore a issue. Maybe can somebody provide some log levels or other debug options for further analysis. Because we are currently hopeless. But I will also try to ask the community.

          Martin Pokorny added a comment - Hi markewaite   That was also my first think, Our proxy is broken or something like that. But now, after restart the Jenkins (docker image) it works as well. Also my curl command works. Therefore a issue. Maybe can somebody provide some log levels or other debug options for further analysis. Because we are currently hopeless. But I will also try to ask the community.

          That was good idea. There are also similar questions, but no body answer it. 

          https://community.jenkins.io/t/jenkins-controller-randomly-losing-tcp-inbound-agent-fixed-port-50000/2346/3

          Therefore I think this is correct place now.

          A hit, maybe it shall be good to configure fixed range of random ports. Not only random or fixed. Because

          + random - no way because nobody want open all ports in the system

          + fixed - because in big systems it can be a gap when all of the agents use the same port.

           

          Martin Pokorny added a comment - That was good idea. There are also similar questions, but no body answer it.  https://community.jenkins.io/t/jenkins-controller-randomly-losing-tcp-inbound-agent-fixed-port-50000/2346/3 Therefore I think this is correct place now. A hit, maybe it shall be good to configure fixed range of random ports. Not only random or fixed. Because + random - no way because nobody want open all ports in the system + fixed - because in big systems it can be a gap when all of the agents use the same port.  

          Andreya Kostov added a comment - - edited

          We are seeing similar behavior after we upgraded to latest LTS - 2.375.1. Our Jenkins instance is not behind a reverse proxy.

          A restart solves the issue, as Martin pointed out. I also found another workaround - changing the port for inbound agents in "Configure Global Security" to a different port than the one we're using, then changing it back. This way we fix the issue without downtime, but it occurs again after some time. Edit: I opened the link to the community forum which gives exactly this workaround after posting this comment.

          I am attaching what I think are relevant logs from our Jenkins instance, with some redacted data - jenkins-agent-connect-error.log.

          Andreya Kostov added a comment - - edited We are seeing similar behavior after we upgraded to latest LTS - 2.375.1. Our Jenkins instance is not behind a reverse proxy. A restart solves the issue, as Martin pointed out. I also found another workaround - changing the port for inbound agents in "Configure Global Security" to a different port than the one we're using, then changing it back. This way we fix the issue without downtime, but it occurs again after some time. Edit: I opened the link to the community forum which gives exactly this workaround after posting this comment. I am attaching what I think are relevant logs from our Jenkins instance, with some redacted data - jenkins-agent-connect-error.log .

          Basil Crow added a comment -

          Duplicates JENKINS-59910.

          Basil Crow added a comment - Duplicates JENKINS-59910 .

          Rahali added a comment -

          Any update for this issue

          Rahali added a comment - Any update for this issue

            Unassigned Unassigned
            mpokornyetm Martin Pokorny
            Votes:
            2 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: