Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46067

Pipeline task scheduled on uninitialized node

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • core, remoting, workflow-durable-task-step-plugin
    • None

      One of our nodes disconnected due to a channel disconnect. Meanwhile a pipeline job was running, and as soon as it reconnected, Jenkins tried to schedule a task on the node, which failed with the error:

       

       

      ERROR: Issue with creating launcher for agent <nodename>. The agent has not been fully initialized yet

      ERROR: Issue with creating launcher for agent <nodename>. The agent has not been fully initialized yet

       

       

      I have attached the console output from the agent.

        1. jenkins.master.trimmed.log
          25 kB
        2. slave.log.1
          8 kB
        3. 2xlarge-738_node.log
          78 kB
        4. catalina_738.log
          65 kB
        5. agentconsole.log
          7 kB

          [JENKINS-46067] Pipeline task scheduled on uninitialized node

          Ben Hines created issue -
          Ben Hines made changes -
          Attachment New: agentconsole.log [ 39195 ]
          Ben Hines made changes -
          Description Original: One of our nodes disconnected due to a channel disconnect. Meanwhile a pipeline job was running, and as soon as it reconnected, Jenkins tried to schedule a task on the node, which failed with the error:

           

           

          ERROR: Issue with creating launcher for agent <nodename>. The agent has not been fully initialized yet

          ERROR: Issue with creating launcher for agent <nodename>. The agent has not been fully initialized yet

                      		New: One of our nodes disconnected due to a channel disconnect. Meanwhile a pipeline job was running, and as soon as it reconnected, Jenkins tried to schedule a task on the node, which failed with the error:

           

           

          ERROR: Issue with creating launcher for agent <nodename>. The agent has not been fully initialized yet

          ERROR: Issue with creating launcher for agent <nodename>. The agent has not been fully initialized yet

           

           

          I have attached the console output from the agent.
          Andrew Bayer made changes -
          Component/s New: workflow-durable-task-step-plugin [ 21715 ]
          Component/s Original: pipeline [ 21692 ]

          Oleg Nenashev added a comment -

          It will be likely fixed by JENKINS-45023. Or you get a failure early at least.

          Oleg Nenashev added a comment - It will be likely fixed by JENKINS-45023 . Or you get a failure early at least.
          Oleg Nenashev made changes -
          Link New: This issue is related to JENKINS-45023 [ JENKINS-45023 ]

          Oleg Nenashev added a comment -

          benh57 Please retest the issue on Remoting 3.14

          Oleg Nenashev added a comment - benh57 Please retest the issue on Remoting 3.14

          John Lengeling added a comment - - edited

          We upgraded to Jenkins LTS 2.89.1 which is supposed to have Remoting 3.14 and we are seeing some builds fail with the same error message.  These are elastic Openstack build nodes.   We are also using a Slave Setup - Setup Script After Copy script too if that matters.
           
          ERROR: Issue with creating launcher for agent tph-build-c1.2xlarge-8428. The agent has not been fully initialized yet
          ERROR: Issue with creating launcher for agent tph-build-c1.2xlarge-8428. The agent has not been fully initialized yet
          remote file operation failed: /opt/jenkins/workspace/fh_guide_master-DAUIRXWDCG4RH3OVIMVYS3W2QDGGAIJIWFEPCAIQEFWE7YG5KZDQ at hudson.remoting.Channel@681ae622:tph-build-c1.2xlarge-8428: hudson.remoting.ChannelClosedException: Remote call on tph-build-c1.2xlarge-8428 failed. The channel is closing down or has closed down
           

          John Lengeling added a comment - - edited We upgraded to Jenkins LTS 2.89.1 which is supposed to have Remoting 3.14 and we are seeing some builds fail with the same error message.  These are elastic Openstack build nodes.   We are also using a Slave Setup - Setup Script After Copy script too if that matters.   ERROR: Issue with creating launcher for agent tph-build-c1.2xlarge-8428. The agent has not been fully initialized yet ERROR: Issue with creating launcher for agent tph-build-c1.2xlarge-8428. The agent has not been fully initialized yet remote file operation failed: /opt/jenkins/workspace/fh_guide_master-DAUIRXWDCG4RH3OVIMVYS3W2QDGGAIJIWFEPCAIQEFWE7YG5KZDQ at hudson.remoting.Channel@681ae622:tph-build-c1.2xlarge-8428: hudson.remoting.ChannelClosedException: Remote call on tph-build-c1.2xlarge-8428 failed. The channel is closing down or has closed down  

          Oleg Nenashev added a comment -

          johnlengeling would you be able to attach...

          1) Jenkins master logs
          2) Agent logs
          3) OpenStack VM provisioniing/termination logs
          4) Agent configuration and the Openstack plugin versions (if any)

          Oleg Nenashev added a comment - johnlengeling would you be able to attach... 1) Jenkins master logs 2) Agent logs 3) OpenStack VM provisioniing/termination logs 4) Agent configuration and the Openstack plugin versions (if any)
          John Lengeling made changes -
          Attachment New: 2xlarge-738_node.log [ 40653 ]
          Attachment New: catalina_738.log [ 40654 ]

            Unassigned Unassigned
            benh57 Ben Hines
            Votes:
            35 Vote for this issue
            Watchers:
            47 Start watching this issue

              Created:
              Updated: