Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-69370

New inbound agent URLs are not accessible by agents (regression in 2.364)

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • core
    • 2.365

      In Jenkins 2.364, when creating a container to perform a build, the JNLP launcher is failing with a 403 forbidden.

      I'm able to recreate the problem with both the Docker and YetAnotherDocker plugins. I'm also able to recreate the problem on the command line.

      Rolling back to Jenkins version 2.363 corrects the issue.

      YetAnotherDocker Output:
      + RUN_CMD='su - jenkins -c '"'"'java -Xmn1G -Xms4G -jar slave.jar -noReconnect -jnlpUrl https://pipeline.#####.com/test//manage/computer/corp%2Djenkins%2Dyad%2Df4ceb4c4b385//slave-agent.jnlp -secret ########################################################'"'"
      + eval 'su - jenkins -c '"'"'java -Xmn1G -Xms4G -jar slave.jar -noReconnect -jnlpUrl https://pipeline.#####.com/test//manage/computer/corp%2Djenkins%2Dyad%2Df4ceb4c4b385//slave-agent.jnlp -secret ########################################################'"'"
      + su - jenkins -c 'java -Xmn1G -Xms4G -jar slave.jar -noReconnect -jnlpUrl https://pipeline.#####.com/test//manage/computer/corp%2Djenkins%2Dyad%2Df4ceb4c4b385//slave-agent.jnlp -secret ########################################################'
      Exception in thread "main" java.io.IOException: Failed to obtain https://pipeline.#####.com/test//manage/computer/corp%2Djenkins%2Dyad%2Df4ceb4c4b385//slave-agent.jnlp?encrypt=true
      at hudson.remoting.Launcher.parseJnlpArguments(Launcher.java:567)
      at hudson.remoting.Launcher.run(Launcher.java:346)
      at hudson.remoting.Launcher.main(Launcher.java:297)
      Caused by: java.io.IOException: Failed to load https://pipeline.#####.com/test//manage/computer/corp%2Djenkins%2Dyad%2Df4ceb4c4b385//slave-agent.jnlp?encrypt=true: 403 Forbidden
      at hudson.remoting.Launcher.parseJnlpArguments(Launcher.java:514)
      ... 2 more

      Command Line:
      bash-5.1# curl -sO https://pipeline.#####.com/test/jnlpJars/agent.jar
      bash-5.1# java -jar agent.jar -jnlpUrl https://pipeline.#####.com/test/manage/computer/jenkins%2Dbc%2Ddid%2D000k7zi0cecag/jenkins-agent.jnlp -secret ########################################################
      Failed to obtain https://pipeline.#####.com/test/manage/computer/jenkins%2Dbc%2Ddid%2D000k7zi0cecag/jenkins-agent.jnlp?encrypt=true
      java.io.IOException: Failed to load https://pipeline.#####.com/test/manage/computer/jenkins%2Dbc%2Ddid%2D000k7zi0cecag/jenkins-agent.jnlp?encrypt=true: 403 Forbidden
      at hudson.remoting.Launcher.parseJnlpArguments(Launcher.java:514)
      at hudson.remoting.Launcher.run(Launcher.java:346)
      at hudson.remoting.Launcher.main(Launcher.java:297)
      Waiting 10 seconds before retry
      Failed to obtain https://pipeline.#####.com/test/manage/computer/jenkins%2Dbc%2Ddid%2D000k7zi0cecag/jenkins-agent.jnlp?encrypt=true
      java.io.IOException: Failed to load https://pipeline.#####.com/test/manage/computer/jenkins%2Dbc%2Ddid%2D000k7zi0cecag/jenkins-agent.jnlp?encrypt=true: 403 Forbidden
      at hudson.remoting.Launcher.parseJnlpArguments(Launcher.java:514)
      at hudson.remoting.Launcher.run(Launcher.java:346)
      at hudson.remoting.Launcher.main(Launcher.java:297)
      Waiting 10 seconds before retry

          [JENKINS-69370] New inbound agent URLs are not accessible by agents (regression in 2.364)

          Basil Crow added a comment -

          What version of agent.jar, and what is the stack trace logged by the controller when the 403 occurs on the agent side?

          Basil Crow added a comment - What version of agent.jar , and what is the stack trace logged by the controller when the 403 occurs on the agent side?

          It would be the same agent.jar that is packaged with jenkins/jenkins:2.364 image.  I believe 3046.v38db_38a_b_7a_86 is also the version packaged with the jenkins/jenkins:2.363 image.

          bash-5.1# java -jar agent.jar  -version
          3046.v38db_38a_b_7a_86 

          What is the stack trace logged by the controller when the 403 occurs on the agent side?
          I don't see any stack traces on the controller when I try to connect.

          When I run the command above:

          **

          bash-5.1# java -jar agent.jar -jnlpUrl https://pipeline######com/test/manage/computer/TEST%2DNODE/jenkins-agent.jnlp -secret ######################################################## -workDir "/tmp/"
          Aug 18, 2022 11:20:15 AM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
          INFO: Using /tmp/remoting as a remoting work directory
          Aug 18, 2022 11:20:15 AM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
          INFO: Both error and output logs will be printed to /tmp/remoting
          Failed to obtain https://pipeline######com/test/manage/computer/TEST%2DNODE/jenkins-agent.jnlp?encrypt=true
          java.io.IOException: Failed to load https://pipeline######com/test/manage/computer/TEST%2DNODE/jenkins-agent.jnlp?encrypt=true: 403 Forbidden
          at hudson.remoting.Launcher.parseJnlpArguments(Launcher.java:514)
          at hudson.remoting.Launcher.run(Launcher.java:346)
          at hudson.remoting.Launcher.main(Launcher.java:297)
          Waiting 10 seconds before retry

          If I roll back to jenkins/jenkins:2.363 and run the connection command above, I can connect successfully.

           

          bash-5.1# java -jar agent.jar -version
          3046.v38db_38a_b_7a_86
          
          bash-5.1# java -jar agent.jar -jnlpUrl https://pipeline######com/test/computer/TEST%2DNODE/jenkins-agent.jnlp -secret ####################################################### -workDir "/tmp/"
          Aug 18, 2022 11:28:19 AM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
          INFO: Using /tmp/remoting as a remoting work directory
          Aug 18, 2022 11:28:19 AM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
          INFO: Both error and output logs will be printed to /tmp/remoting
          Aug 18, 2022 11:28:19 AM hudson.remoting.jnlp.Main createEngine
          INFO: Setting up agent: TEST-NODE
          Aug 18, 2022 11:28:19 AM hudson.remoting.Engine startEngine
          INFO: Using Remoting version: 3046.v38db_38a_b_7a_86
          Aug 18, 2022 11:28:19 AM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
          INFO: Using /tmp/remoting as a remoting work directory
          Aug 18, 2022 11:28:19 AM hudson.remoting.jnlp.Main$CuiListener status
          INFO: Locating server among [https://pipeline######com/test/]
          Aug 18, 2022 11:28:19 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
          INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
          Aug 18, 2022 11:28:19 AM hudson.remoting.jnlp.Main$CuiListener status
          INFO: Agent discovery successful
          Agent address: pipeline######com
          Agent port: 50007
          Identity: 99:12:d8:fd:fd:ba:1f:38:a9:dc:a9:ac:87:91:96:fe
          Aug 18, 2022 11:28:19 AM hudson.remoting.jnlp.Main$CuiListener status
          INFO: Handshaking
          Aug 18, 2022 11:28:19 AM hudson.remoting.jnlp.Main$CuiListener status
          INFO: Connecting to pipeline######com:50007
          Aug 18, 2022 11:28:19 AM hudson.remoting.jnlp.Main$CuiListener status
          INFO: Trying protocol: JNLP4-connect
          Aug 18, 2022 11:28:19 AM org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader run
          INFO: Waiting for ProtocolStack to start.
          Aug 18, 2022 11:28:20 AM hudson.remoting.jnlp.Main$CuiListener status
          INFO: Remote identity confirmed: 99:12:d8:fd:fd:ba:1f:38:a9:dc:a9:ac:87:91:96:fe
          Aug 18, 2022 11:28:20 AM hudson.remoting.jnlp.Main$CuiListener status
          INFO: Connected

           

          William Gillaspy added a comment - It would be the same agent.jar that is packaged with jenkins/jenkins:2.364 image.  I believe 3046.v38db_38a_b_7a_86 is also the version packaged with the jenkins/jenkins:2.363 image. bash-5.1# java -jar agent.jar  -version 3046.v38db_38a_b_7a_86 What is the stack trace logged by the controller when the 403 occurs on the agent side? I don't see any stack traces on the controller when I try to connect. When I run the command above: ** bash-5.1# java -jar agent.jar -jnlpUrl https: //pipeline######com/test/manage/computer/TEST%2DNODE/jenkins-agent.jnlp -secret ######################################################## -workDir "/tmp/" Aug 18, 2022 11:20:15 AM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir INFO: Using /tmp/remoting as a remoting work directory Aug 18, 2022 11:20:15 AM org.jenkinsci.remoting.engine.WorkDirManager setupLogging INFO: Both error and output logs will be printed to /tmp/remoting Failed to obtain https: //pipeline######com/test/manage/computer/TEST%2DNODE/jenkins-agent.jnlp?encrypt= true java.io.IOException: Failed to load https: //pipeline######com/test/manage/computer/TEST%2DNODE/jenkins-agent.jnlp?encrypt= true : 403 Forbidden at hudson.remoting.Launcher.parseJnlpArguments(Launcher.java:514) at hudson.remoting.Launcher.run(Launcher.java:346) at hudson.remoting.Launcher.main(Launcher.java:297) Waiting 10 seconds before retry If I roll back to jenkins/jenkins:2.363 and run the connection command above, I can connect successfully.   bash-5.1# java -jar agent.jar -version 3046.v38db_38a_b_7a_86 bash-5.1# java -jar agent.jar -jnlpUrl https: //pipeline######com/test/computer/TEST%2DNODE/jenkins-agent.jnlp -secret ####################################################### -workDir "/tmp/" Aug 18, 2022 11:28:19 AM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir INFO: Using /tmp/remoting as a remoting work directory Aug 18, 2022 11:28:19 AM org.jenkinsci.remoting.engine.WorkDirManager setupLogging INFO: Both error and output logs will be printed to /tmp/remoting Aug 18, 2022 11:28:19 AM hudson.remoting.jnlp.Main createEngine INFO: Setting up agent: TEST-NODE Aug 18, 2022 11:28:19 AM hudson.remoting.Engine startEngine INFO: Using Remoting version: 3046.v38db_38a_b_7a_86 Aug 18, 2022 11:28:19 AM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir INFO: Using /tmp/remoting as a remoting work directory Aug 18, 2022 11:28:19 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Locating server among [https: //pipeline######com/test/] Aug 18, 2022 11:28:19 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping] Aug 18, 2022 11:28:19 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Agent discovery successful Agent address: pipeline######com Agent port: 50007 Identity: 99:12:d8:fd:fd:ba:1f:38:a9:dc:a9:ac:87:91:96:fe Aug 18, 2022 11:28:19 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Handshaking Aug 18, 2022 11:28:19 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Connecting to pipeline######com:50007 Aug 18, 2022 11:28:19 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Trying protocol: JNLP4-connect Aug 18, 2022 11:28:19 AM org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader run INFO: Waiting for ProtocolStack to start. Aug 18, 2022 11:28:20 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Remote identity confirmed: 99:12:d8:fd:fd:ba:1f:38:a9:dc:a9:ac:87:91:96:fe Aug 18, 2022 11:28:20 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Connected  

          Basil Crow added a comment -

          I don't see any stack traces on the controller when I try to connect.

          Not even in the controller's log file? The UI generally doesn't show internal errors.

          If you can run git bisect on the Jenkins core commits between 2.363 and 2.364 to identify the commit that introduced the problem, that would help.

          If you can't do that and want us to do more investigation, you need to provide far more detailed steps to reproduce the problem from scratch. I've never used this Docker plugin. Please provide instructions that assume nothing, including installing Jenkins and your set of plugins from scratch against a clean Jenkins home, and show how to duplicate the problem. Then others might be able to assist you better.

          Basil Crow added a comment - I don't see any stack traces on the controller when I try to connect. Not even in the controller's log file? The UI generally doesn't show internal errors. If you can run git bisect on the Jenkins core commits between 2.363 and 2.364 to identify the commit that introduced the problem, that would help. If you can't do that and want us to do more investigation, you need to provide far more detailed steps to reproduce the problem from scratch. I've never used this Docker plugin. Please provide instructions that assume nothing, including installing Jenkins and your set of plugins from scratch against a clean Jenkins home, and show how to duplicate the problem. Then others might be able to assist you better.

          basil, thank you for the reply.  I'm able to create this issue with any jnlp agent connection to a 2.364 jenkins image, not just the plugins initially listed in my issue.

          I can recreate the issue on a brand new Jenkins with the default plugin configuration.

           

          docker run --name=test-jenkins -d -p 8080:8080 -p 50000:50000 jenkins/jenkins:2.364
          docker logs test-jenkins

          <open the logs and grab the admin password to log in>

           

          1. Open browser to the instance, enter admin password
          2. Install suggested plugins.
          3. Wait, then create first admin user
          4. Set jenkins url to your workstation ip and exposed port
          5. Log into jenkins with the first admin user.
          6. Click Manage Jenkins, Manage Nodes and Clouds
          7. Click New Node
          8. Set name to Test-Node, click the radio for permanent agent.
          9. Set root directory to /tmp
          10. Launch method should be "Launch agent by connecting it to the controller
          11. Click Save
          12. Click "Test Node" on the node list.
          13. Copy the Run agent command line information.

          Now start a jdk container:

          docker run -it openjdk:11 bash

          Paste the agent command line into the bash prompt.

           

           

          root@3969ac32c836:/# curl -sO http://10.1.0.33:8080/jnlpJars/agent.jar
          root@3969ac32c836:/# java -jar agent.jar -jnlpUrl http://10.1.0.33:8080/manage/computer/TEST%2DNODE/jenkins-agent.jnlp -secret 60c3051162bbe57df0c80bbeaca1e427860093e633e383541661cb8a18a3112d -workDir ""

          Press enter.

          Receive the error:

           

           

           

           

          Aug 18, 2022 4:54:27 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
          INFO: Using /remoting as a remoting work directory
          Aug 18, 2022 4:54:27 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
          INFO: Both error and output logs will be printed to /remoting
          Failed to obtain http://10.1.0.33:8080/manage/computer/TEST%2DNODE/jenkins-agent.jnlp?encrypt=true
          java.io.IOException: Failed to load http://10.1.0.33:8080/manage/computer/TEST%2DNODE/jenkins-agent.jnlp?encrypt=true: 403 Forbidden
                  at hudson.remoting.Launcher.parseJnlpArguments(Launcher.java:514)
                  at hudson.remoting.Launcher.run(Launcher.java:346)
                  at hudson.remoting.Launcher.main(Launcher.java:297)
          Waiting 10 seconds before retry

           

          Check the Jenkins log and see that there are no stack traces.

          If you repeat the process with the jenkins/jenkins:2.363 image, you won't receive the 403 error and the agent will connect.

          I've looked a the diff between version and I don't see anything that sticks out as causing the problem.  The change log mentions that Winstone was updated.  Would that cause this behavior?

          Thank you for your assistance,

          William

          William Gillaspy added a comment - basil , thank you for the reply.  I'm able to create this issue with any jnlp agent connection to a 2.364 jenkins image, not just the plugins initially listed in my issue. I can recreate the issue on a brand new Jenkins with the default plugin configuration.   docker run --name=test-jenkins -d -p 8080:8080 -p 50000:50000 jenkins/jenkins:2.364 docker logs test-jenkins <open the logs and grab the admin password to log in>   Open browser to the instance, enter admin password Install suggested plugins. Wait, then create first admin user Set jenkins url to your workstation ip and exposed port Log into jenkins with the first admin user. Click Manage Jenkins, Manage Nodes and Clouds Click New Node Set name to Test-Node, click the radio for permanent agent. Set root directory to /tmp Launch method should be "Launch agent by connecting it to the controller Click Save Click "Test Node" on the node list. Copy the Run agent command line information. Now start a jdk container: docker run -it openjdk:11 bash Paste the agent command line into the bash prompt.     root@3969ac32c836:/# curl -sO http: //10.1.0.33:8080/jnlpJars/agent.jar root@3969ac32c836:/# java -jar agent.jar -jnlpUrl http: //10.1.0.33:8080/manage/computer/TEST%2DNODE/jenkins-agent.jnlp -secret 60c3051162bbe57df0c80bbeaca1e427860093e633e383541661cb8a18a3112d -workDir "" Press enter. Receive the error:         Aug 18, 2022 4:54:27 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir INFO: Using /remoting as a remoting work directory Aug 18, 2022 4:54:27 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging INFO: Both error and output logs will be printed to /remoting Failed to obtain http: //10.1.0.33:8080/manage/computer/TEST%2DNODE/jenkins-agent.jnlp?encrypt= true java.io.IOException: Failed to load http: //10.1.0.33:8080/manage/computer/TEST%2DNODE/jenkins-agent.jnlp?encrypt= true : 403 Forbidden         at hudson.remoting.Launcher.parseJnlpArguments(Launcher.java:514)         at hudson.remoting.Launcher.run(Launcher.java:346)         at hudson.remoting.Launcher.main(Launcher.java:297) Waiting 10 seconds before retry   Check the Jenkins log and see that there are no stack traces. If you repeat the process with the jenkins/jenkins:2.363 image, you won't receive the 403 error and the agent will connect. I've looked a the diff between version and I don't see anything that sticks out as causing the problem.  The change log mentions that Winstone was updated.  Would that cause this behavior? Thank you for your assistance, William

          Basil Crow added a comment -

          Bisection shows this was caused by jenkinsci/jenkins#6907 (CC janfaracik). This regression affects the connection of inbound agents. In the absence of a verified fix by the end of the week, I plan to revert the offending PR prior to next week's release of 2.365.

          Basil Crow added a comment - Bisection shows this was caused by jenkinsci/jenkins#6907 (CC janfaracik ). This regression affects the connection of inbound agents. In the absence of a verified fix by the end of the week, I plan to revert the offending PR prior to next week's release of 2.365.

          Daniel Beck added a comment -

          As a workaround, you can edit the agent command line arguments as follows:

          The web page will provide:

          java -jar agent.jar -jnlpUrl http://localhost:8080/manage/computer/(agentName)/jenkins-agent.jnlp 

          Remove manage/ from the URL:

          java -jar agent.jar -jnlpUrl http://localhost:8080/computer/(agentName)/jenkins-agent.jnlp 

           

          Daniel Beck added a comment - As a workaround, you can edit the agent command line arguments as follows: The web page will provide: java -jar agent.jar -jnlpUrl http://localhost:8080/manage/computer/(agentName)/jenkins-agent.jnlp Remove manage/ from the URL: java -jar agent.jar -jnlpUrl http://localhost:8080/computer/(agentName)/jenkins-agent.jnlp  

            timja Tim Jacomb
            wgillaspy William Gillaspy
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: