Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-9117

Jenkins hangs in the shutdown mode after thinBackup is triggered

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • thinbackup-plugin
    • None

      thinBackup triggers a backup job and did not return from the shutdown mode

          [JENKINS-9117] Jenkins hangs in the shutdown mode after thinBackup is triggered

          Frank Merrow added a comment - - edited

          Yeah, this is a show stopper for us . . . I cannot have Jenkins shutting itself down.

          I thought at first perhaps the issue was related to using a UNC path \\<server>\<share>\somepath, but mounting the share as a network drive and using x:\somepath didn't help.

          Account HAS ACCESS to this location, I've browsed there as that user, created and deleted a file. (Jenkins itself has yet to write so much as one byte in the specified directory).

          First four settings: X:\Users\hjenkins\March-Web, 30 0 * * *, 10 * * * *, 10

          All other settings are blank or unchecked.

          Note this works fine for me at home no issues . . . the biggest difference at work is that the number of jobs is of course much bigger (over 100) and the number of plugins.

          Just in case it is a plugin interaction issue . . . plugin info below:

          Maven 2 Project Plugin
          Jenkin's Maven 2 project type
          Warning: This plugin is built for Jenkins 1.452 or newer. It may or may not work in your Jenkins.
          1.452 1.450

          Static Code Analysis Plug-ins
          This plug-in provides utilities for the static code analysis plug-ins.
          1.38 1.29

          Warnings Plugin
          This plugin generates the trend report for compiler warnings in the console log or in log files.
          3.28 3.21

          Green Balls
          Changes Hudson to use green balls instead of blue for successful builds
          1.11 1.10

          Perforce Plugin
          Integrates Jenkins with Perforce SCM Repositories.
          1.3.9 1.3.8-SNAPSHOT (private-01/23/2012 01:19-gregf)

          CVS Plugin
          This bundled plugin integrates Jenkins with CVS version control system.
          2.0 1.6

          Text-finder Plugin
          This plugin lets you search keywords in the files you specified and use that to downgrade a "successful" build to be unstable or a failure.
          1.9 1.8

          Subversion Plugin
          This plugin adds the Subversion support (via SVNKit) to Jenkins.
          1.39 1.34

          Groovy plugin
          This plugin adds the ability to directly execute Groovy code.
          1.11 1.10

          Scriptler Plugin
          Scriptler allows you to store/edit groovy scripts and execute it on any of the slaves/nodes... no need for copy paste groovy code anymore.

          Frank Merrow added a comment - - edited Yeah, this is a show stopper for us . . . I cannot have Jenkins shutting itself down. I thought at first perhaps the issue was related to using a UNC path \\<server>\<share>\somepath, but mounting the share as a network drive and using x:\somepath didn't help. Account HAS ACCESS to this location, I've browsed there as that user, created and deleted a file. (Jenkins itself has yet to write so much as one byte in the specified directory). First four settings: X:\Users\hjenkins\March-Web, 30 0 * * *, 10 * * * *, 10 All other settings are blank or unchecked. Note this works fine for me at home no issues . . . the biggest difference at work is that the number of jobs is of course much bigger (over 100) and the number of plugins. Just in case it is a plugin interaction issue . . . plugin info below: Maven 2 Project Plugin Jenkin's Maven 2 project type Warning: This plugin is built for Jenkins 1.452 or newer. It may or may not work in your Jenkins. 1.452 1.450 Static Code Analysis Plug-ins This plug-in provides utilities for the static code analysis plug-ins. 1.38 1.29 Warnings Plugin This plugin generates the trend report for compiler warnings in the console log or in log files. 3.28 3.21 Green Balls Changes Hudson to use green balls instead of blue for successful builds 1.11 1.10 Perforce Plugin Integrates Jenkins with Perforce SCM Repositories. 1.3.9 1.3.8-SNAPSHOT (private-01/23/2012 01:19-gregf) CVS Plugin This bundled plugin integrates Jenkins with CVS version control system. 2.0 1.6 Text-finder Plugin This plugin lets you search keywords in the files you specified and use that to downgrade a "successful" build to be unstable or a failure. 1.9 1.8 Subversion Plugin This plugin adds the Subversion support (via SVNKit) to Jenkins. 1.39 1.34 Groovy plugin This plugin adds the ability to directly execute Groovy code. 1.11 1.10 Scriptler Plugin Scriptler allows you to store/edit groovy scripts and execute it on any of the slaves/nodes... no need for copy paste groovy code anymore.

          Thomas Fürer added a comment -

          Yes, I do not need to shutdown jenkins. The intention why I set jenkins in a shutdown mode is to provide a save state where no configurations, results, ... can be changed during a running job, for the backup itself.
          I decided to scipt this need for the next release.

          Thanks for your feedback.

          Thomas Fürer added a comment - Yes, I do not need to shutdown jenkins. The intention why I set jenkins in a shutdown mode is to provide a save state where no configurations, results, ... can be changed during a running job, for the backup itself. I decided to scipt this need for the next release. Thanks for your feedback.

          Thomas Fürer added a comment -

          @Frank
          You do a FULL backup every day half an hour past midnight and do a DIFF backup 10 minutes past every hour. Correct?
          I don't know how fast your network is this could be in some cases the cause why he stays in the shutdown-mode. Please try to set another schedule which runs not concurrent and not so many times, to try to isolate the problem.

          For example my configuration is:

          • FULL: 0 23 * * 0 #every Sunday 23:00
          • DIFF: 30 19 * * 1-5 #every weekday 19:30

          I have a Ubuntu 11.10 and a local backup disk

          @Mingjiang Shi
          Interessting informations are the logs, the system configuration, number of jobs, aproximatly backup size, thinbackup configuration
          do you also have so tight schedules? if yes please olso try to do the same as I suggest to Frank.

          Thanks for your help!

          Thomas Fürer added a comment - @Frank You do a FULL backup every day half an hour past midnight and do a DIFF backup 10 minutes past every hour. Correct? I don't know how fast your network is this could be in some cases the cause why he stays in the shutdown-mode. Please try to set another schedule which runs not concurrent and not so many times, to try to isolate the problem. For example my configuration is: FULL: 0 23 * * 0 #every Sunday 23:00 DIFF: 30 19 * * 1-5 #every weekday 19:30 I have a Ubuntu 11.10 and a local backup disk @Mingjiang Shi Interessting informations are the logs, the system configuration, number of jobs, aproximatly backup size, thinbackup configuration do you also have so tight schedules? if yes please olso try to do the same as I suggest to Frank. Thanks for your help!

          Frank Merrow added a comment -

          So I've managed to duplicate this issue on a really small Jenkins running on my Laptop . . . like 10 jobs, and all those hand started, basically an idle system with limited history . . . I'm not sure what the backup size is, but all I see it a bunch of 1K files . . . it cannot be much.

          It looks like there are some exceptions in the Jenkins log . . . I will upload it next. Nice that I was able to duplicate it in just in this limited environment, is it very unlikely there is anything company proprietary here . . . so I can post freely.

          Frank

          Frank Merrow added a comment - So I've managed to duplicate this issue on a really small Jenkins running on my Laptop . . . like 10 jobs, and all those hand started, basically an idle system with limited history . . . I'm not sure what the backup size is, but all I see it a bunch of 1K files . . . it cannot be much. It looks like there are some exceptions in the Jenkins log . . . I will upload it next. Nice that I was able to duplicate it in just in this limited environment, is it very unlikely there is anything company proprietary here . . . so I can post freely. Frank

          Thomas Fürer added a comment -

          @Frank
          the exceptions in the log are not related to the thinbackup. I could be the HTMLPublisher, maybe the Warnings or the Greenballs plugin.

          Just to be shure, the red bar, saying jenkins will shutdown, will not removed after backup is finished on your systems, also if you refresh the browser?

          It would be interessting which OS you choose.

          Thomas Fürer added a comment - @Frank the exceptions in the log are not related to the thinbackup. I could be the HTMLPublisher, maybe the Warnings or the Greenballs plugin. Just to be shure, the red bar, saying jenkins will shutdown, will not removed after backup is finished on your systems, also if you refresh the browser? It would be interessting which OS you choose.

          Frank Merrow added a comment -

          Windows . . .

          And yes! It took a few minutes, but if I kept hitting refresh eventually the "shutdown" message went away!!! It was the first time I had seen this, but then typically I don't hit refresh . . .

          So . . . I get where you are coming from wanting a stable system to backup . . . but this seems like a pretty heavy handed way to get it. It is a pretty shocking thing for a user to see who isn't in the know . . . and if they have the ability, they are likely to cancel on you anyway.

          Plus on my "almost nothing" Jenkins the backup took 1 minute or so . . . on our big production Jenkins with all the data and history, it will be interesting to see how long it lasts . . . it could be awhile.

          Clearly this is upsetting (to users) enough that it will have to be done at night, but we are a world wide 7/24 operation. I have a smaller community to upset in Hyderabad than I do in San Diego, but it is hard to select a backup time that won't alarm one of them. Plus night time is the time for "big builds" . . . the multi-hour things nobody want to do during the day . . . I am reluctant to take a big chunk of time out of that already busy schedule . . . no easy solutions here.

          It might be interesting to see if KK would add a new "mode" . . . shutdown like in that tables/files don't change, but instead of a big red banner saying shutdown, a (say) blue banner saying "backup in progress" . . . something to clue the user into the "real situation".

          Thanks for looking into this . . . sorry the solution is "I just didn't understand" . . .

          Frank

          Frank Merrow added a comment - Windows . . . And yes! It took a few minutes, but if I kept hitting refresh eventually the "shutdown" message went away!!! It was the first time I had seen this, but then typically I don't hit refresh . . . So . . . I get where you are coming from wanting a stable system to backup . . . but this seems like a pretty heavy handed way to get it. It is a pretty shocking thing for a user to see who isn't in the know . . . and if they have the ability, they are likely to cancel on you anyway. Plus on my "almost nothing" Jenkins the backup took 1 minute or so . . . on our big production Jenkins with all the data and history, it will be interesting to see how long it lasts . . . it could be awhile. Clearly this is upsetting (to users) enough that it will have to be done at night, but we are a world wide 7/24 operation. I have a smaller community to upset in Hyderabad than I do in San Diego, but it is hard to select a backup time that won't alarm one of them. Plus night time is the time for "big builds" . . . the multi-hour things nobody want to do during the day . . . I am reluctant to take a big chunk of time out of that already busy schedule . . . no easy solutions here. It might be interesting to see if KK would add a new "mode" . . . shutdown like in that tables/files don't change, but instead of a big red banner saying shutdown, a (say) blue banner saying "backup in progress" . . . something to clue the user into the "real situation". Thanks for looking into this . . . sorry the solution is "I just didn't understand" . . . Frank

          Thomas Fürer added a comment -

          @frank
          can I close this issue now?

          i have removed the shutdown state from the backup. only if you restore the system you need the shutdown mode.

          Thomas Fürer added a comment - @frank can I close this issue now? i have removed the shutdown state from the backup. only if you restore the system you need the shutdown mode.

          Frank Merrow added a comment -

          Well, there are several folks on this ticket (in fact I am not the creator), but I feel the situation has been explained (and even fixed given backup won't cause the issue only restore will). (Note I looked and saw 1.5 was available, I installed it and verified the problem does not exist in that version.)

          With the caveat that I am not the created, it is okay with me if you close the ticket.

          THANK YOU!

          Frank

          Frank Merrow added a comment - Well, there are several folks on this ticket (in fact I am not the creator), but I feel the situation has been explained (and even fixed given backup won't cause the issue only restore will). (Note I looked and saw 1.5 was available, I installed it and verified the problem does not exist in that version.) With the caveat that I am not the created, it is okay with me if you close the ticket. THANK YOU! Frank

          Thomas Fürer added a comment -

          fixed in 1.5
          additionaly jenkins is not put in the shutdown mode for the backup anymore

          Thomas Fürer added a comment - fixed in 1.5 additionaly jenkins is not put in the shutdown mode for the backup anymore

          Code changed in jenkins
          User: tofuatgit
          Path:
          src/main/java/org/jvnet/hudson/plugins/thinbackup/ThinBackupPeriodicWork.java
          http://jenkins-ci.org/commit/thin-backup-plugin/6d363ab67abf2260d8d759d06f690fb564b3e3b9
          Log:
          fix JENKINS-9117: Jenkins hangs in the shutdown mode after thinBackup is triggered

          git-svn-id: https://svn.jenkins-ci.org/trunk/hudson/plugins/thinBackup@39091 71c3de6d-444a-0410-be80-ed276b4c234a

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: tofuatgit Path: src/main/java/org/jvnet/hudson/plugins/thinbackup/ThinBackupPeriodicWork.java http://jenkins-ci.org/commit/thin-backup-plugin/6d363ab67abf2260d8d759d06f690fb564b3e3b9 Log: fix JENKINS-9117 : Jenkins hangs in the shutdown mode after thinBackup is triggered git-svn-id: https://svn.jenkins-ci.org/trunk/hudson/plugins/thinBackup@39091 71c3de6d-444a-0410-be80-ed276b4c234a

            tofuatjava Thomas Fürer
            tofuatjava Thomas Fürer
            Votes:
            2 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: