Auto retry for elastic agents after channel closure

This issue is archived. You can view it, but you can't modify it. Learn more

XMLWordPrintable

      While my pipeline was running, the node that was executing logic terminated. I see this at the bottom of my console output:

      Cannot contact ip-172-31-242-8.us-west-2.compute.internal: java.io.IOException: remote file operation failed: /ebs/jenkins/workspace/common-pipelines-nodeploy at hudson.remoting.Channel@48503f20:ip-172-31-242-8.us-west-2.compute.internal: hudson.remoting.ChannelClosedException: Channel "unknown": Remote call on ip-172-31-242-8.us-west-2.compute.internal failed. The channel is closing down or has closed down

      There's a spinning arrow below it.

      I have a cron script that uses the Jenkins master CLI to remove nodes which have stopped responding. When I examine this node's page in my Jenkins website, it looks like the node is still running that job and i see an orange label that says "Feb 22, 2018 5:16:02 PM Node is being removed".

      I'm wondering what would be a better way to say "If the channel closes down, retry the work on another node with the same label?

      Things seem stuck. Please advise.

        1. Thread dump [Jenkins].html
          219 kB
        2. threadDump.txt
          98 kB
        3. support_2018-07-04_07.35.22.zip
          956 kB
        4. slaveLogInMaster.grub.zip
          8 kB
        5. NetworkAndMachineStats.png
          NetworkAndMachineStats.png
          224 kB
        6. MonitoringJavaelodyOnNodes.html
          44 kB
        7. jobConsoleOutput.txt
          12 kB
        8. jobConsoleOutput.txt
          12 kB
        9. jenkins.log
          984 kB
        10. jenkins_support_2018-06-29_01.14.18.zip
          1.26 MB
        11. jenkins_agents_Thread_dump.html
          172 kB
        12. jenkins_Agent_devbuild9_System_Information.html
          66 kB
        13. jenkins_agent_devbuild9_remoting_logs.zip
          4 kB
        14. JavaMelodyNodeGrubThreads_4_07_18.pdf
          9 kB
        15. JavaMelodyGrubHeapDump_4_07_18.pdf
          220 kB
        16. image-2018-02-22-17-28-03-053.png
          image-2018-02-22-17-28-03-053.png
          30 kB
        17. image-2018-02-22-17-27-31-541.png
          image-2018-02-22-17-27-31-541.png
          56 kB
        18. grubSystemInformation.html
          67 kB
        19. grub.remoting.logs.zip
          3 kB

            Assignee:
            Jesse Glick
            Reporter:
            Jon B
            Archiver:
            Jenkins Service Account

              Created:
              Updated:
              Resolved:
              Archived: