Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-49707

Auto retry for elastic agents after channel closure

    XMLWordPrintable

Details

    Description

      While my pipeline was running, the node that was executing logic terminated. I see this at the bottom of my console output:

      Cannot contact ip-172-31-242-8.us-west-2.compute.internal: java.io.IOException: remote file operation failed: /ebs/jenkins/workspace/common-pipelines-nodeploy at hudson.remoting.Channel@48503f20:ip-172-31-242-8.us-west-2.compute.internal: hudson.remoting.ChannelClosedException: Channel "unknown": Remote call on ip-172-31-242-8.us-west-2.compute.internal failed. The channel is closing down or has closed down

      There's a spinning arrow below it.

      I have a cron script that uses the Jenkins master CLI to remove nodes which have stopped responding. When I examine this node's page in my Jenkins website, it looks like the node is still running that job and i see an orange label that says "Feb 22, 2018 5:16:02 PM Node is being removed".

      I'm wondering what would be a better way to say "If the channel closes down, retry the work on another node with the same label?

      Things seem stuck. Please advise.

      Attachments

        1. grub.remoting.logs.zip
          3 kB
        2. grubSystemInformation.html
          67 kB
        3. image-2018-02-22-17-27-31-541.png
          image-2018-02-22-17-27-31-541.png
          56 kB
        4. image-2018-02-22-17-28-03-053.png
          image-2018-02-22-17-28-03-053.png
          30 kB
        5. JavaMelodyGrubHeapDump_4_07_18.pdf
          220 kB
        6. JavaMelodyNodeGrubThreads_4_07_18.pdf
          9 kB
        7. jenkins_agent_devbuild9_remoting_logs.zip
          4 kB
        8. jenkins_Agent_devbuild9_System_Information.html
          66 kB
        9. jenkins_agents_Thread_dump.html
          172 kB
        10. jenkins_support_2018-06-29_01.14.18.zip
          1.26 MB
        11. jenkins.log
          984 kB
        12. jobConsoleOutput.txt
          12 kB
        13. jobConsoleOutput.txt
          12 kB
        14. MonitoringJavaelodyOnNodes.html
          44 kB
        15. NetworkAndMachineStats.png
          NetworkAndMachineStats.png
          224 kB
        16. slaveLogInMaster.grub.zip
          8 kB
        17. support_2018-07-04_07.35.22.zip
          956 kB
        18. threadDump.txt
          98 kB
        19. Thread dump [Jenkins].html
          219 kB

        Issue Links

          causes

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-69936 PWD returning wrong path

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-70528 node / dir / node on same agent sets PWD to that of dir rather than @2 workspace

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          depends on

          Improvement - An improvement or enhancement to an existing feature or task. JENKINS-30383 SynchronousNonBlockingStepExecution should allow restart of idempotent steps

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-49241 pipeline hangs if slave node momentarily disconnects

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-47868 Pipeline durability hang when slave node disconnected

          • Major - Major loss of function.
          • Reopened

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-43781 Quickly detecting and restarting a job if the job's slave disconnects

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-57675 Pipeline steps running forever when executor fails

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-47561 Pipelines wait indefinitely for kubernetes slaves to come back online

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. JENKINS-43607 Jenkins pipeline not aborted when the machine running docker container goes offline

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. JENKINS-56673 Better handling of ChannelClosedException in Declarative pipeline

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-41854 Contextualize a fresh FilePath after an agent reconnection

          • Major - Major loss of function.
          • Resolved
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. JENKINS-36013 Automatically abort ExecutorPickle rehydration from an ephemeral node

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-61387 SlaveComputer not cleaned up after the channel is closed

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-67285 if jenkins-agent pod has removed fail fast jobs that use this jenkins-agent pod

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-71113 AgentErrorCondition should handle "missing workspace" error

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-59340 Pipeline hangs when Agent pod is Terminated

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-60507 Pipeline stuck when allocating machine | node block appears to be neither running nor scheduled

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-35246 Kubernetes agents not getting deleted in Jenkins after pods are deleted

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. JENKINS-70333 Default for Declarative agent retries

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. JENKINS-68963 build logs should contain if a spot agent is terminated

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open
          links to

          Web Link CloudBees-internal issue

          Web Link jenkins-infra/docker-jenkins-weekly #512

          Web Link jenkins-infra/pipeline-library #405

          Web Link kubernetes-plugin #1083

          Web Link pipeline-model-definition-plugin #533

          Web Link workflow-api-plugin #217

          Web Link workflow-basic-steps-plugin #195

          Web Link workflow-durable-task-step-plugin #180

          Web Link workflow-durable-task-step-plugin #254

          Activity

            People

              jglick Jesse Glick
              piratejohnny Jon B
              Votes:
              37 Vote for this issue
              Watchers:
              54 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: