Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-9117

Jenkins hangs in the shutdown mode after thinBackup is triggered

    XMLWordPrintable

Details

    Description

      thinBackup triggers a backup job and did not return from the shutdown mode

      Attachments

        Issue Links

          Activity

            tofuatjava Thomas Fürer added a comment -

            will be fixed with thinBackup 1.3

            tofuatjava Thomas Fürer added a comment - will be fixed with thinBackup 1.3
            dtka Daniel Tkatch added a comment -

            I have 1.4 installed and the problem still occurs.

            dtka Daniel Tkatch added a comment - I have 1.4 installed and the problem still occurs.
            tofuatjava Thomas Fürer added a comment -

            can you provide me more information about you system configuration. we cannot reproduce such a case.

            tofuatjava Thomas Fürer added a comment - can you provide me more information about you system configuration. we cannot reproduce such a case.
            mjshi Mingjiang Shi added a comment -

            I have 1.4 installed. The problem occurs when a manual full backup is triggered via the web interface. Not sure this happened when automatic full backup is started.

            mjshi Mingjiang Shi added a comment - I have 1.4 installed. The problem occurs when a manual full backup is triggered via the web interface. Not sure this happened when automatic full backup is started.
            mjshi Mingjiang Shi added a comment -

            @Thomas Fürer,
            If a time-consuming job is triggered, the message "Jenkins is going to shutdown" will be displayed. After I disabled the plugin, the message is not displayed any more.

            Please let me know what information you need to diagnose the issue. This is a very annoying issue.

            mjshi Mingjiang Shi added a comment - @Thomas Fürer, If a time-consuming job is triggered, the message "Jenkins is going to shutdown" will be displayed. After I disabled the plugin, the message is not displayed any more. Please let me know what information you need to diagnose the issue. This is a very annoying issue.

            Hi all,

            actually we also face the same problem with our Jenkins instance.
            The version of the plugin we use is 1.4 also.
            This is strange, as I think that when I read about the plugin I understood it does not shut down the Jenkins instance, right?

            Regards,
            Ralitsa

            ralitsa_chipeva Ralitsa Chipeva added a comment - Hi all, actually we also face the same problem with our Jenkins instance. The version of the plugin we use is 1.4 also. This is strange, as I think that when I read about the plugin I understood it does not shut down the Jenkins instance, right? Regards, Ralitsa
            fmerrow Frank Merrow added a comment - - edited

            Yeah, this is a show stopper for us . . . I cannot have Jenkins shutting itself down.

            I thought at first perhaps the issue was related to using a UNC path \\<server>\<share>\somepath, but mounting the share as a network drive and using x:\somepath didn't help.

            Account HAS ACCESS to this location, I've browsed there as that user, created and deleted a file. (Jenkins itself has yet to write so much as one byte in the specified directory).

            First four settings: X:\Users\hjenkins\March-Web, 30 0 * * *, 10 * * * *, 10

            All other settings are blank or unchecked.

            Note this works fine for me at home no issues . . . the biggest difference at work is that the number of jobs is of course much bigger (over 100) and the number of plugins.

            Just in case it is a plugin interaction issue . . . plugin info below:

            Maven 2 Project Plugin
            Jenkin's Maven 2 project type
            Warning: This plugin is built for Jenkins 1.452 or newer. It may or may not work in your Jenkins.
            1.452 1.450

            Static Code Analysis Plug-ins
            This plug-in provides utilities for the static code analysis plug-ins.
            1.38 1.29

            Warnings Plugin
            This plugin generates the trend report for compiler warnings in the console log or in log files.
            3.28 3.21

            Green Balls
            Changes Hudson to use green balls instead of blue for successful builds
            1.11 1.10

            Perforce Plugin
            Integrates Jenkins with Perforce SCM Repositories.
            1.3.9 1.3.8-SNAPSHOT (private-01/23/2012 01:19-gregf)

            CVS Plugin
            This bundled plugin integrates Jenkins with CVS version control system.
            2.0 1.6

            Text-finder Plugin
            This plugin lets you search keywords in the files you specified and use that to downgrade a "successful" build to be unstable or a failure.
            1.9 1.8

            Subversion Plugin
            This plugin adds the Subversion support (via SVNKit) to Jenkins.
            1.39 1.34

            Groovy plugin
            This plugin adds the ability to directly execute Groovy code.
            1.11 1.10

            Scriptler Plugin
            Scriptler allows you to store/edit groovy scripts and execute it on any of the slaves/nodes... no need for copy paste groovy code anymore.

            fmerrow Frank Merrow added a comment - - edited Yeah, this is a show stopper for us . . . I cannot have Jenkins shutting itself down. I thought at first perhaps the issue was related to using a UNC path \\<server>\<share>\somepath, but mounting the share as a network drive and using x:\somepath didn't help. Account HAS ACCESS to this location, I've browsed there as that user, created and deleted a file. (Jenkins itself has yet to write so much as one byte in the specified directory). First four settings: X:\Users\hjenkins\March-Web, 30 0 * * *, 10 * * * *, 10 All other settings are blank or unchecked. Note this works fine for me at home no issues . . . the biggest difference at work is that the number of jobs is of course much bigger (over 100) and the number of plugins. Just in case it is a plugin interaction issue . . . plugin info below: Maven 2 Project Plugin Jenkin's Maven 2 project type Warning: This plugin is built for Jenkins 1.452 or newer. It may or may not work in your Jenkins. 1.452 1.450 Static Code Analysis Plug-ins This plug-in provides utilities for the static code analysis plug-ins. 1.38 1.29 Warnings Plugin This plugin generates the trend report for compiler warnings in the console log or in log files. 3.28 3.21 Green Balls Changes Hudson to use green balls instead of blue for successful builds 1.11 1.10 Perforce Plugin Integrates Jenkins with Perforce SCM Repositories. 1.3.9 1.3.8-SNAPSHOT (private-01/23/2012 01:19-gregf) CVS Plugin This bundled plugin integrates Jenkins with CVS version control system. 2.0 1.6 Text-finder Plugin This plugin lets you search keywords in the files you specified and use that to downgrade a "successful" build to be unstable or a failure. 1.9 1.8 Subversion Plugin This plugin adds the Subversion support (via SVNKit) to Jenkins. 1.39 1.34 Groovy plugin This plugin adds the ability to directly execute Groovy code. 1.11 1.10 Scriptler Plugin Scriptler allows you to store/edit groovy scripts and execute it on any of the slaves/nodes... no need for copy paste groovy code anymore.
            tofuatjava Thomas Fürer added a comment -

            Yes, I do not need to shutdown jenkins. The intention why I set jenkins in a shutdown mode is to provide a save state where no configurations, results, ... can be changed during a running job, for the backup itself.
            I decided to scipt this need for the next release.

            Thanks for your feedback.

            tofuatjava Thomas Fürer added a comment - Yes, I do not need to shutdown jenkins. The intention why I set jenkins in a shutdown mode is to provide a save state where no configurations, results, ... can be changed during a running job, for the backup itself. I decided to scipt this need for the next release. Thanks for your feedback.
            tofuatjava Thomas Fürer added a comment -

            @Frank
            You do a FULL backup every day half an hour past midnight and do a DIFF backup 10 minutes past every hour. Correct?
            I don't know how fast your network is this could be in some cases the cause why he stays in the shutdown-mode. Please try to set another schedule which runs not concurrent and not so many times, to try to isolate the problem.

            For example my configuration is:

            • FULL: 0 23 * * 0 #every Sunday 23:00
            • DIFF: 30 19 * * 1-5 #every weekday 19:30

            I have a Ubuntu 11.10 and a local backup disk

            @Mingjiang Shi
            Interessting informations are the logs, the system configuration, number of jobs, aproximatly backup size, thinbackup configuration
            do you also have so tight schedules? if yes please olso try to do the same as I suggest to Frank.

            Thanks for your help!

            tofuatjava Thomas Fürer added a comment - @Frank You do a FULL backup every day half an hour past midnight and do a DIFF backup 10 minutes past every hour. Correct? I don't know how fast your network is this could be in some cases the cause why he stays in the shutdown-mode. Please try to set another schedule which runs not concurrent and not so many times, to try to isolate the problem. For example my configuration is: FULL: 0 23 * * 0 #every Sunday 23:00 DIFF: 30 19 * * 1-5 #every weekday 19:30 I have a Ubuntu 11.10 and a local backup disk @Mingjiang Shi Interessting informations are the logs, the system configuration, number of jobs, aproximatly backup size, thinbackup configuration do you also have so tight schedules? if yes please olso try to do the same as I suggest to Frank. Thanks for your help!
            fmerrow Frank Merrow added a comment -

            So I've managed to duplicate this issue on a really small Jenkins running on my Laptop . . . like 10 jobs, and all those hand started, basically an idle system with limited history . . . I'm not sure what the backup size is, but all I see it a bunch of 1K files . . . it cannot be much.

            It looks like there are some exceptions in the Jenkins log . . . I will upload it next. Nice that I was able to duplicate it in just in this limited environment, is it very unlikely there is anything company proprietary here . . . so I can post freely.

            Frank

            fmerrow Frank Merrow added a comment - So I've managed to duplicate this issue on a really small Jenkins running on my Laptop . . . like 10 jobs, and all those hand started, basically an idle system with limited history . . . I'm not sure what the backup size is, but all I see it a bunch of 1K files . . . it cannot be much. It looks like there are some exceptions in the Jenkins log . . . I will upload it next. Nice that I was able to duplicate it in just in this limited environment, is it very unlikely there is anything company proprietary here . . . so I can post freely. Frank
            tofuatjava Thomas Fürer added a comment -

            @Frank
            the exceptions in the log are not related to the thinbackup. I could be the HTMLPublisher, maybe the Warnings or the Greenballs plugin.

            Just to be shure, the red bar, saying jenkins will shutdown, will not removed after backup is finished on your systems, also if you refresh the browser?

            It would be interessting which OS you choose.

            tofuatjava Thomas Fürer added a comment - @Frank the exceptions in the log are not related to the thinbackup. I could be the HTMLPublisher, maybe the Warnings or the Greenballs plugin. Just to be shure, the red bar, saying jenkins will shutdown, will not removed after backup is finished on your systems, also if you refresh the browser? It would be interessting which OS you choose.
            fmerrow Frank Merrow added a comment -

            Windows . . .

            And yes! It took a few minutes, but if I kept hitting refresh eventually the "shutdown" message went away!!! It was the first time I had seen this, but then typically I don't hit refresh . . .

            So . . . I get where you are coming from wanting a stable system to backup . . . but this seems like a pretty heavy handed way to get it. It is a pretty shocking thing for a user to see who isn't in the know . . . and if they have the ability, they are likely to cancel on you anyway.

            Plus on my "almost nothing" Jenkins the backup took 1 minute or so . . . on our big production Jenkins with all the data and history, it will be interesting to see how long it lasts . . . it could be awhile.

            Clearly this is upsetting (to users) enough that it will have to be done at night, but we are a world wide 7/24 operation. I have a smaller community to upset in Hyderabad than I do in San Diego, but it is hard to select a backup time that won't alarm one of them. Plus night time is the time for "big builds" . . . the multi-hour things nobody want to do during the day . . . I am reluctant to take a big chunk of time out of that already busy schedule . . . no easy solutions here.

            It might be interesting to see if KK would add a new "mode" . . . shutdown like in that tables/files don't change, but instead of a big red banner saying shutdown, a (say) blue banner saying "backup in progress" . . . something to clue the user into the "real situation".

            Thanks for looking into this . . . sorry the solution is "I just didn't understand" . . .

            Frank

            fmerrow Frank Merrow added a comment - Windows . . . And yes! It took a few minutes, but if I kept hitting refresh eventually the "shutdown" message went away!!! It was the first time I had seen this, but then typically I don't hit refresh . . . So . . . I get where you are coming from wanting a stable system to backup . . . but this seems like a pretty heavy handed way to get it. It is a pretty shocking thing for a user to see who isn't in the know . . . and if they have the ability, they are likely to cancel on you anyway. Plus on my "almost nothing" Jenkins the backup took 1 minute or so . . . on our big production Jenkins with all the data and history, it will be interesting to see how long it lasts . . . it could be awhile. Clearly this is upsetting (to users) enough that it will have to be done at night, but we are a world wide 7/24 operation. I have a smaller community to upset in Hyderabad than I do in San Diego, but it is hard to select a backup time that won't alarm one of them. Plus night time is the time for "big builds" . . . the multi-hour things nobody want to do during the day . . . I am reluctant to take a big chunk of time out of that already busy schedule . . . no easy solutions here. It might be interesting to see if KK would add a new "mode" . . . shutdown like in that tables/files don't change, but instead of a big red banner saying shutdown, a (say) blue banner saying "backup in progress" . . . something to clue the user into the "real situation". Thanks for looking into this . . . sorry the solution is "I just didn't understand" . . . Frank
            tofuatjava Thomas Fürer added a comment -

            @frank
            can I close this issue now?

            i have removed the shutdown state from the backup. only if you restore the system you need the shutdown mode.

            tofuatjava Thomas Fürer added a comment - @frank can I close this issue now? i have removed the shutdown state from the backup. only if you restore the system you need the shutdown mode.
            fmerrow Frank Merrow added a comment -

            Well, there are several folks on this ticket (in fact I am not the creator), but I feel the situation has been explained (and even fixed given backup won't cause the issue only restore will). (Note I looked and saw 1.5 was available, I installed it and verified the problem does not exist in that version.)

            With the caveat that I am not the created, it is okay with me if you close the ticket.

            THANK YOU!

            Frank

            fmerrow Frank Merrow added a comment - Well, there are several folks on this ticket (in fact I am not the creator), but I feel the situation has been explained (and even fixed given backup won't cause the issue only restore will). (Note I looked and saw 1.5 was available, I installed it and verified the problem does not exist in that version.) With the caveat that I am not the created, it is okay with me if you close the ticket. THANK YOU! Frank
            tofuatjava Thomas Fürer added a comment -

            fixed in 1.5
            additionaly jenkins is not put in the shutdown mode for the backup anymore

            tofuatjava Thomas Fürer added a comment - fixed in 1.5 additionaly jenkins is not put in the shutdown mode for the backup anymore

            Code changed in jenkins
            User: tofuatgit
            Path:
            src/main/java/org/jvnet/hudson/plugins/thinbackup/ThinBackupPeriodicWork.java
            http://jenkins-ci.org/commit/thin-backup-plugin/6d363ab67abf2260d8d759d06f690fb564b3e3b9
            Log:
            fix JENKINS-9117: Jenkins hangs in the shutdown mode after thinBackup is triggered

            git-svn-id: https://svn.jenkins-ci.org/trunk/hudson/plugins/thinBackup@39091 71c3de6d-444a-0410-be80-ed276b4c234a

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: tofuatgit Path: src/main/java/org/jvnet/hudson/plugins/thinbackup/ThinBackupPeriodicWork.java http://jenkins-ci.org/commit/thin-backup-plugin/6d363ab67abf2260d8d759d06f690fb564b3e3b9 Log: fix JENKINS-9117 : Jenkins hangs in the shutdown mode after thinBackup is triggered git-svn-id: https://svn.jenkins-ci.org/trunk/hudson/plugins/thinBackup@39091 71c3de6d-444a-0410-be80-ed276b4c234a

            People

              tofuatjava Thomas Fürer
              tofuatjava Thomas Fürer
              Votes:
              2 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: