Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-54502

Jenkins crashed when a running build was aborted.

      After recently updating to version 2.149 or 2.150, aborting a running build no matter automatically or manually resulted in Jenkins crashed itself. 

      In jenkins,err.log I couldn't find any useful/related log. Could anyone please help to investigate this annoying issue?

       

      Environment: Windows 2012 R2

          [JENKINS-54502] Jenkins crashed when a running build was aborted.

          Vipul Madaan added a comment - - edited

          I am having the exact same issue on version 2.150 . Any update on it?

          Vipul Madaan added a comment - - edited I am having the exact same issue on version 2.150 . Any update on it?

          Scott Jiang added a comment -

          No. Still frustrated.

          Scott Jiang added a comment - No. Still frustrated.

          marlene cote added a comment -

          happening to me too. please fix this.

          marlene cote added a comment - happening to me too. please fix this.

          I am facing the exact same issue as well (ver:2.150). Any one found a workaround yet? 

          Yatharth Zutshi added a comment - I am facing the exact same issue as well (ver:2.150). Any one found a workaround yet? 

          The same thing happened to me since I updated jenkins from version 2.138.3 to 2.182. When launching a compilation using the windows command step and aborting the task jenkins crashes. I've fixed it by commenting the calls to the "killSoftly" method in "ProcessTree.java".

           
          diff --git "a/core/src/main/java/hudson/util/ProcessTree.java" "b/core/src/main/java/hudson/util/ProcessTree.java"
          index 4e2a4b3c14..98a8cf04ea 100644
          — "a/core/src/main/java/hudson/util/ProcessTree.java"
          +++ "b/core/src/main/java/hudson/util/ProcessTree.java"
          @@ -26,7 +26,6 @@ package hudson.util;
          import com.sun.jna.Memory;
          import com.sun.jna.Native;
          import com.sun.jna.NativeLong;
          -import com.sun.jna.Pointer;
          import com.sun.jna.LastErrorException;
          import com.sun.jna.ptr.IntByReference;
          import hudson.EnvVars;
          @@ -520,7 +519,7 @@ public abstract class ProcessTree implements Iterable<OSProcess>, IProcessTree,

          LOGGER.log(FINER, "Killing recursively {0}", getPid());
          // Firstly try to kill the root process gracefully, then do a forcekill if it does not help (algorithm is described in JENKINS-17116)

          • killSoftly();
            + // killSoftly();
            p.killRecursively();
            killByKiller();
            }
            @@ -533,7 +532,7 @@ public abstract class ProcessTree implements Iterable<OSProcess>, IProcessTree,

          LOGGER.log(FINER, "Killing {0}", getPid());
          // Firstly try to kill it gracefully, then do a forcekill if it does not help (algorithm is described in JENKINS-17116)

          • killSoftly();
            + // killSoftly();
            p.kill();
            killByKiller();
            }
            @@ -1442,7 +1441,7 @@ public abstract class ProcessTree implements Iterable<OSProcess>, IProcessTree,
          • Read the remainder of psinfo_t differently depending on whether the
          • Java process is 32-bit or 64-bit.
            */
          • if (Pointer.SIZE == 8) {
            + if (Native.POINTER_SIZE == 8) {
            psinfo.seek(236); // offset of pr_argc
            argc = adjust(psinfo.readInt());
            argp = adjustL(psinfo.readLong());

          Jose Antonio Castelbón added a comment - The same thing happened to me since I updated jenkins from version 2.138.3 to 2.182. When launching a compilation using the windows command step and aborting the task jenkins crashes. I've fixed it by commenting the calls to the "killSoftly" method in "ProcessTree.java".   diff --git "a/core/src/main/java/hudson/util/ProcessTree.java" "b/core/src/main/java/hudson/util/ProcessTree.java" index 4e2a4b3c14..98a8cf04ea 100644 — "a/core/src/main/java/hudson/util/ProcessTree.java" +++ "b/core/src/main/java/hudson/util/ProcessTree.java" @@ -26,7 +26,6 @@ package hudson.util; import com.sun.jna.Memory; import com.sun.jna.Native; import com.sun.jna.NativeLong; -import com.sun.jna.Pointer; import com.sun.jna.LastErrorException; import com.sun.jna.ptr.IntByReference; import hudson.EnvVars; @@ -520,7 +519,7 @@ public abstract class ProcessTree implements Iterable<OSProcess>, IProcessTree, LOGGER.log(FINER, "Killing recursively {0}", getPid()); // Firstly try to kill the root process gracefully, then do a forcekill if it does not help (algorithm is described in JENKINS-17116 ) killSoftly(); + // killSoftly(); p.killRecursively(); killByKiller(); } @@ -533,7 +532,7 @@ public abstract class ProcessTree implements Iterable<OSProcess>, IProcessTree, LOGGER.log(FINER, "Killing {0}", getPid()); // Firstly try to kill it gracefully, then do a forcekill if it does not help (algorithm is described in JENKINS-17116 ) killSoftly(); + // killSoftly(); p.kill(); killByKiller(); } @@ -1442,7 +1441,7 @@ public abstract class ProcessTree implements Iterable<OSProcess>, IProcessTree, Read the remainder of psinfo_t differently depending on whether the Java process is 32-bit or 64-bit. */ if (Pointer.SIZE == 8) { + if (Native.POINTER_SIZE == 8) { psinfo.seek(236); // offset of pr_argc argc = adjust(psinfo.readInt()); argp = adjustL(psinfo.readLong());

          Owen Mehegan added a comment -

          If jacastelbon is correct about the fix, the change was introduced at https://github.com/jenkinsci/jenkins/pull/3414. vipulmadaan brought this issue to my attention yesterday. I asked him to reproduce it and capture a support bundle from Jenkins to attach to this issue. Interestingly, he said that if he installs a recent version of Jenkins from scratch, he can't reproduce this issue. He only sees it after upgrading. Not sure why that would make a difference in the abort functionality. He is running his builds on the master, which I suspect is relevant here; perhaps aborting the job is somehow killing the master java process. That would explain why nothing is logged when people see this.

          Owen Mehegan added a comment - If jacastelbon is correct about the fix, the change was introduced at https://github.com/jenkinsci/jenkins/pull/3414 . vipulmadaan brought this issue to my attention yesterday. I asked him to reproduce it and capture a support bundle from Jenkins to attach to this issue. Interestingly, he said that if he installs a recent version of Jenkins from scratch, he can't reproduce this issue. He only sees it after upgrading. Not sure why that would make a difference in the abort functionality. He is running his builds on the master, which I suspect is relevant here; perhaps aborting the job is somehow killing the master java process. That would explain why nothing is logged when people see this.

          owenmehegan, could it be that the WinP library isn't properly upgraded? If a fresh install works alright but an upgrade doesn't, it may be a problem with updates of libraries.

          oleg_nenashev ..?

          Stephan Reiter added a comment - owenmehegan , could it be that the WinP library isn't properly upgraded? If a fresh install works alright but an upgrade doesn't, it may be a problem with updates of libraries. oleg_nenashev ..?

          Owen Mehegan added a comment -

          sreiter that's a good question. I actually have no personal experience running Jenkins on Windows, I just wanted to add some context to this Jira after meeting vipulmadaan at Jenkins World last week.

          Vipul, am I remembering correctly that you have been upgrading Jenkins by just downloading the newer jar file and putting it in place? Did you originally use a Windows installer package to install Jenkins? Maybe this is related.

          Owen Mehegan added a comment - sreiter that's a good question. I actually have no personal experience running Jenkins on Windows, I just wanted to add some context to this Jira after meeting vipulmadaan at Jenkins World last week. Vipul, am I remembering correctly that you have been upgrading Jenkins by just downloading the newer jar file and putting it in place? Did you originally use a Windows installer package to install Jenkins? Maybe this is related.

          Vipul Madaan added a comment -

          Thank you sreiter and owenmehegan

          I have tried placing a new jar file and also tried to upgrade using Jenkins GUI under manage jenkins. Had same issue both times.

          I am not sure how it was originally installed. Is there a way to check it?

           

          Also I will be trying again this week and will update the crash logs here.

           

          Vipul Madaan added a comment - Thank you sreiter and owenmehegan I have tried placing a new jar file and also tried to upgrade using Jenkins GUI under manage jenkins. Had same issue both times. I am not sure how it was originally installed. Is there a way to check it?   Also I will be trying again this week and will update the crash logs here.  

          Owen Mehegan added a comment -

          vipulmadaan manually downloading the jar and using the upgrade button in the Jenkins UI are effectively the same thing.

          This page shows a few ways that you can list installed MSI packages on Windows: https://www.makeuseof.com/tag/list-installed-programs-windows/ I would try this and see if Jenkins is in this list. If so, that would indicate that the Windows package was used to install Jenkins. If that is the case, I would recommend that you use the Windows package to upgrade it. You can get it from here: https://jenkins.io/download/

          Owen Mehegan added a comment - vipulmadaan manually downloading the jar and using the upgrade button in the Jenkins UI are effectively the same thing. This page shows a few ways that you can list installed MSI packages on Windows: https://www.makeuseof.com/tag/list-installed-programs-windows/ I would try this and see if Jenkins is in this list. If so, that would indicate that the Windows package was used to install Jenkins. If that is the case, I would recommend that you use the Windows package to upgrade it. You can get it from here: https://jenkins.io/download/

          Vipul Madaan added a comment -

          Yes I can see jenkins in list of installed package.

          So now if I use jenkins.msi to upgrade, do i need to follow any special instructions or just installing it in same location will upgrade it.

           

          Thanks

          Vipul

          Vipul Madaan added a comment - Yes I can see jenkins in list of installed package. So now if I use jenkins.msi to upgrade, do i need to follow any special instructions or just installing it in same location will upgrade it.   Thanks Vipul

          Owen Mehegan added a comment -

          vipulmadaan I can't say for certain, having never run Jenkins on Windows. I see no reason why just running the installer wouldn't work. I expect the difference between the installer and the bare jar file is that the installer sets up the Windows service, puts the jar in the right place, and includes other Windows-specific things like the WinP library.

          Owen Mehegan added a comment - vipulmadaan I can't say for certain, having never run Jenkins on Windows. I see no reason why just running the installer wouldn't work. I expect the difference between the installer and the bare jar file is that the installer sets up the Windows service, puts the jar in the right place, and includes other Windows-specific things like the WinP library.

          Keith Davis added a comment - - edited

          I'm on 2.189 and believe I have the same issue - aborting running jobs causes the service to crash. Does not happen 100% of the time (I don't think, but have not confirmed that part). 

           

          Child process [2064 - c:\Jenkins\jre\bin\java -Xrs -Xmx256m -Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle -jar "c:\Jenkins\jenkins.war" --httpPort=9999] finished with -1073741510

           

          The jenkins service terminated unexpectedly. It has done this 1 time(s). The following corrective action will be taken in 60000 milliseconds: Restart the service.

           

           

           

          Keith Davis added a comment - - edited I'm on 2.189 and believe I have the same issue - aborting running jobs causes the service to crash. Does not happen 100% of the time (I don't think, but have not confirmed that part).    Child process [2064 - c:\Jenkins\jre\bin\java -Xrs -Xmx256m -Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle -jar "c:\Jenkins\jenkins.war" --httpPort=9999] finished with -1073741510   The jenkins service terminated unexpectedly. It has done this 1 time(s). The following corrective action will be taken in 60000 milliseconds: Restart the service.      

          We are experiencing the same problem on a x64 system.

          I noticed that winp.dll and sendctrlc.exe of the WinP library are extracted to the <jenkins>\war\WEB-INF\lib directory under some long name like winp.E336920A48E5EB5F70FF7D65DA840462.dll.
          These however are not the x64 versions, but the 32 bit ones.

          I extracted the winp-1.28.jar file, replaced winp.dll with the content of winp.x64.dll and did the same for the exe file. So after that there is no chance for WinP to use the wrong version, because both are 64 bit now. I re-packaged the jar and placed it in <jenkins>\war\WEB-INF\lib.

          With this setup aborting jobs no longer kills Jenkins. However I also no longer see extracted versions of the dll and exe files in the lib directory.

          Konrad Kügler added a comment - We are experiencing the same problem on a x64 system. I noticed that winp.dll and sendctrlc.exe of the WinP library are extracted to the <jenkins>\war\WEB-INF\lib directory under some long name like winp.E336920A48E5EB5F70FF7D65DA840462.dll. These however are not the x64 versions, but the 32 bit ones. I extracted the winp-1.28.jar file, replaced winp.dll with the content of winp.x64.dll and did the same for the exe file. So after that there is no chance for WinP to use the wrong version, because both are 64 bit now. I re-packaged the jar and placed it in <jenkins>\war\WEB-INF\lib. With this setup aborting jobs no longer kills Jenkins. However I also no longer see extracted versions of the dll and exe files in the lib directory.

          I'm seeing the same issue in the Linux environment (RHEL6). The crashes have become very frequent after upgrading. Is there a workaround/fix 

          Abhishek Sharma added a comment - I'm seeing the same issue in the Linux environment (RHEL6). The crashes have become very frequent after upgrading. Is there a workaround/fix 

          Owen Mehegan added a comment -

          abhsha if you are having this issue on Linux, it is most likely a different bug. I would suggest creating a new issue, and attach a support bundle with logs if you can. Use the Support Core plugin to generate a bundle.

          Owen Mehegan added a comment - abhsha if you are having this issue on Linux, it is most likely a different bug. I would suggest creating a new issue, and attach a support bundle with logs if you can. Use the Support Core plugin to generate a bundle.

          Is this still an issue? I'm also seeing this behaviour on 2.204.1

          ingolfur Thorisson added a comment - Is this still an issue? I'm also seeing this behaviour on 2.204.1

          Vipul Madaan added a comment -

          Hello

          My issue got resolved . I downloaded the Executable of Jenkins and instead of running an upgrade from UI Or replacing a jar file Idid use newly downloaded executable to do a fresh install of newer version on top of my previous version . So now I have newer version and don’t have any crashes . I have tested it for a while and I am not able to reproduce the crash .

          Vipul Madaan added a comment - Hello My issue got resolved . I downloaded the Executable of Jenkins and instead of running an upgrade from UI Or replacing a jar file Idid use newly downloaded executable to do a fresh install of newer version on top of my previous version . So now I have newer version and don’t have any crashes . I have tested it for a while and I am not able to reproduce the crash .

          That fixed it for me as well. Thanks for the help Vipul!

          ingolfur Thorisson added a comment - That fixed it for me as well. Thanks for the help Vipul!

            Unassigned Unassigned
            scottjx Scott Jiang
            Votes:
            8 Vote for this issue
            Watchers:
            15 Start watching this issue

              Created:
              Updated:
              Resolved: