Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-25704

Scheduled triggering stops working until restart

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Critical Critical
    • core

      Jobs configured with a periodic trigger stop being triggered after Jenkins has been running for a few days.

      There are no indications of a problem in the log. The last/next run times shown in the configure page look right. Manually triggering the job works, though the periodic trigger doesn't resume until Jenkins is restarted.

      The Jenkins and plugin versions are what is currently installed though we've seen the problem for some months with different recent versions of Jenkins and various plugins. It might be that the problem has not been noticed more often because of more frequent Jenkins restarts.

      I've attached a thread dump taken while periodic triggers haven't been working for over a day.

        1. jenkins thread dumps.txt
          13 kB
        2. threadDump.txt
          38 kB
        3. jenkins_log_threaddump-tom.txt
          65 kB
        4. hudson.triggers-ALL.log
          42 kB
        5. hudson.triggers-ALL.log
          42 kB
        6. thread-dump1.txt
          67 kB

          [JENKINS-25704] Scheduled triggering stops working until restart

          Daniel Beck added a comment -

          Could you search the Jenkins log for "Timer task foo failed", foo being the name of a task?

          Daniel Beck added a comment - Could you search the Jenkins log for "Timer task foo failed", foo being the name of a task?

          Daniel Beck added a comment -

          Could you set up a log recorder for the logger hudson.triggers.Trigger on level FINE to see what gets logged?

          https://wiki.jenkins-ci.org/display/JENKINS/Logging

          Daniel Beck added a comment - Could you set up a log recorder for the logger hudson.triggers.Trigger on level FINE to see what gets logged? https://wiki.jenkins-ci.org/display/JENKINS/Logging

          David Resnick added a comment -

          The problem has finally recurred after I configured the logger. The logs show that the trigger just stops at some point.

          This is one of the cron jobs:

          Nov 26, 2014 2:36:00 AM FINE hudson.triggers.Trigger
cron checking hudson.model.FreeStyleProject@4f4e362a[Reviewbot2] with spec ‘H/5 * * * *’
Nov 26, 2014 2:37:00 AM FINE hudson.triggers.Trigger
cron checking hudson.model.FreeStyleProject@4f4e362a[Reviewbot2] with spec ‘H/5 * * * *’
Nov 26, 2014 2:38:00 AM FINE hudson.triggers.Trigger
cron checking hudson.model.FreeStyleProject@4f4e362a[Reviewbot2] with spec ‘H/5 * * * *’
Nov 26, 2014 2:38:00 AM CONFIG hudson.triggers.Trigger
cron triggered hudson.model.FreeStyleProject@4f4e362a[Reviewbot2]
Nov 26, 2014 2:39:00 AM FINE hudson.triggers.Trigger
cron checking hudson.model.FreeStyleProject@4f4e362a[Reviewbot2] with spec ‘H/5 * * * *’
Nov 26, 2014 2:40:00 AM FINE hudson.triggers.Trigger
cron checking hudson.model.FreeStyleProject@4f4e362a[Reviewbot2] with spec ‘H/5 * * * *’
Nov 26, 2014 2:41:00 AM FINE hudson.triggers.Trigger
cron checking hudson.model.FreeStyleProject@4f4e362a[Reviewbot2] with spec ‘H/5 * * * *’

          David Resnick added a comment - The problem has finally recurred after I configured the logger. The logs show that the trigger just stops at some point. This is one of the cron jobs: Nov 26, 2014 2:36:00 AM FINE hudson.triggers.Trigger cron checking hudson.model.FreeStyleProject@4f4e362a[Reviewbot2] with spec ‘H/5 * * * *’ Nov 26, 2014 2:37:00 AM FINE hudson.triggers.Trigger cron checking hudson.model.FreeStyleProject@4f4e362a[Reviewbot2] with spec ‘H/5 * * * *’ Nov 26, 2014 2:38:00 AM FINE hudson.triggers.Trigger cron checking hudson.model.FreeStyleProject@4f4e362a[Reviewbot2] with spec ‘H/5 * * * *’ Nov 26, 2014 2:38:00 AM CONFIG hudson.triggers.Trigger cron triggered hudson.model.FreeStyleProject@4f4e362a[Reviewbot2] Nov 26, 2014 2:39:00 AM FINE hudson.triggers.Trigger cron checking hudson.model.FreeStyleProject@4f4e362a[Reviewbot2] with spec ‘H/5 * * * *’ Nov 26, 2014 2:40:00 AM FINE hudson.triggers.Trigger cron checking hudson.model.FreeStyleProject@4f4e362a[Reviewbot2] with spec ‘H/5 * * * *’ Nov 26, 2014 2:41:00 AM FINE hudson.triggers.Trigger cron checking hudson.model.FreeStyleProject@4f4e362a[Reviewbot2] with spec ‘H/5 * * * *’

          Daniel Beck added a comment -

          Are you changing date/time on the Jenkins master, e.g. during tests?

          Daniel Beck added a comment - Are you changing date/time on the Jenkins master, e.g. during tests?

          David Resnick added a comment - - edited

          No. AFAIK there are no changes taking place in the environment. The last time triggers stopped was in the middle of the night.

          David Resnick added a comment - - edited No. AFAIK there are no changes taking place in the environment. The last time triggers stopped was in the middle of the night.

          David Resnick added a comment - - edited

          No building is done on the master, though a couple jobs do run there (e.g. the Reviewbot2 one in the log above).

          David Resnick added a comment - - edited No building is done on the master, though a couple jobs do run there (e.g. the Reviewbot2 one in the log above).

          Daniel Beck added a comment -

          Are you running an NTP client?

          Daniel Beck added a comment - Are you running an NTP client?

          David Resnick added a comment -

          I'll be able to check tomorrow. I did check some logs on the master today (e.g. cron) and saw that there were no jumps of the date/time. Timestamps in jenkins.log are also predictable, after the triggers stop too.

          David Resnick added a comment - I'll be able to check tomorrow. I did check some logs on the master today (e.g. cron) and saw that there were no jumps of the date/time. Timestamps in jenkins.log are also predictable, after the triggers stop too.

          David Resnick added a comment -

          ntpd is running on the server. It seems to be functioning OK. Time on the server is synced, ntpd hasn't logged anything.

          David Resnick added a comment - ntpd is running on the server. It seems to be functioning OK. Time on the server is synced, ntpd hasn't logged anything.

          Kanstantsin Shautsou added a comment -

          Try enable more logs. Enable logging for "hudson.triggers" with ALL level.

          Kanstantsin Shautsou added a comment - Try enable more logs. Enable logging for "hudson.triggers" with ALL level.

          David Resnick added a comment -

          I've attached log hudson.triggers-ALL.log. It seems to show that again, for no apparent reason, cron triggers stop at a certain point.

          David Resnick added a comment - I've attached log hudson.triggers-ALL.log. It seems to show that again, for no apparent reason, cron triggers stop at a certain point.

          hans kuykens added a comment -

          We are experiencing a similar issue with Jenkins V 1.590
          This happens on a server with two additional nodes.

          hans kuykens added a comment - We are experiencing a similar issue with Jenkins V 1.590 This happens on a server with two additional nodes.

          Jonathan Strickland added a comment -

          Finding this issue with 1.621 ... https://issues.jenkins-ci.org/browse/JENKINS-30558. When viewing some of our git polling jobs we see that they are no longer showing as polling anymore. We poll every 5 minutes for these jobs and found they were 8+ hours from last poll time.

          Jonathan Strickland added a comment - Finding this issue with 1.621 ... https://issues.jenkins-ci.org/browse/JENKINS-30558 . When viewing some of our git polling jobs we see that they are no longer showing as polling anymore. We poll every 5 minutes for these jobs and found they were 8+ hours from last poll time.

          Jonathan Strickland added a comment -

          Before I restart it; I have tried to turn on fine level for the hudson.trigger.Trigger and hudson.trigger.Trigger$Cron logs; nothing coming in. I plan to leave this logger on after reload.

          Jonathan Strickland added a comment - Before I restart it; I have tried to turn on fine level for the hudson.trigger.Trigger and hudson.trigger.Trigger$Cron logs; nothing coming in. I plan to leave this logger on after reload.

          Daniel Beck added a comment -

          Check the /threadDump URL for a "Jenkins cron thread" – what is it doing when the triggers stop working?

          Daniel Beck added a comment - Check the /threadDump URL for a "Jenkins cron thread" – what is it doing when the triggers stop working?

          Tom Lachner added a comment -

          I have the same issue and according to attached thread-dump1.txt it looks like the following:

          "Jenkins cron thread" prio=5 TIMED_WAITING
	java.lang.Object.wait(Native Method)
	java.util.TimerThread.mainLoop(Timer.java:552)
	java.util.TimerThread.run(Timer.java:505)

          For me it looked like this when the issue appears:

          "Jenkins cron thread" Id=19 Group=main WAITING on java.util.TaskQueue@73054d75
	at java.lang.Object.wait(Native Method)
	-  waiting on java.util.TaskQueue@73054d75
	at java.lang.Object.wait(Object.java:503)
	at java.util.TimerThread.mainLoop(Timer.java:526)
	at java.util.TimerThread.run(Timer.java:505)

          Could be the same issue as in JENKINS-30558 because we have also the stash plugin installed and I can not remember same kind of issues before stash plugin.

          Tom Lachner added a comment - I have the same issue and according to attached thread-dump1.txt it looks like the following: "Jenkins cron thread" prio=5 TIMED_WAITING java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) For me it looked like this when the issue appears: "Jenkins cron thread" Id=19 Group=main WAITING on java.util.TaskQueue@73054d75 at java.lang.Object.wait(Native Method) - waiting on java.util.TaskQueue@73054d75 at java.lang.Object.wait(Object.java:503) at java.util.TimerThread.mainLoop(Timer.java:526) at java.util.TimerThread.run(Timer.java:505) Could be the same issue as in JENKINS-30558 because we have also the stash plugin installed and I can not remember same kind of issues before stash plugin.

          David Resnick added a comment -

          I didn't encounter this problem when I had the stash plugin installed. I now have the stash plugin and haven't encountered this problem.

          David Resnick added a comment - I didn't encounter this problem when I had the stash plugin installed. I now have the stash plugin and haven't encountered this problem.

          Jonathan Strickland added a comment - - edited

          timguy See in your thread dump if you see that of a thread dump at JENKINS-30588

          java.net.SocketInputStream.socketRead0
          ....
          followed in stack is the stashpullrequest timer

          Its misleading but yes the cron thread may be waiting on completion of the timer task of the stash pull request, which will never complete due to a hung socket. david_resnick, we actually didn't see the issue until multiple teams in our group starting having multiple repos each with pull requests (thus multiple jobs with more load on our production stash instance)

          Jonathan Strickland added a comment - - edited timguy See in your thread dump if you see that of a thread dump at JENKINS-30588 java.net.SocketInputStream.socketRead0 .... followed in stack is the stashpullrequest timer Its misleading but yes the cron thread may be waiting on completion of the timer task of the stash pull request, which will never complete due to a hung socket. david_resnick , we actually didn't see the issue until multiple teams in our group starting having multiple repos each with pull requests (thus multiple jobs with more load on our production stash instance)

          Tom Lachner added a comment -

          jwstric2 Well, I see the socketRead0 and something with stash in the thread dump. I attached my thread dump jenkins_log_threaddump-tom.txt and it looks quite similar like in ticket JENKINS-30558. Thanks

          Tom Lachner added a comment - jwstric2 Well, I see the socketRead0 and something with stash in the thread dump. I attached my thread dump jenkins_log_threaddump-tom.txt and it looks quite similar like in ticket JENKINS-30558 . Thanks

          Tomas D added a comment -

          We just found the same problem. We have pretty much nothing running on our new Jenkins server, created new pipeline. Manually it runs fine, but it failed to run last night. Schedule is set to 

          H 2 * * *

          I am attaching threadDump threadDump.txt

          Tomas D added a comment - We just found the same problem. We have pretty much nothing running on our new Jenkins server, created new pipeline. Manually it runs fine, but it failed to run last night. Schedule is set to H 2 * * * I am attaching threadDump threadDump.txt

          Sekhar Babu added a comment -

          I am also facing the same problem from past 1 week.

          Jenkins scheduler is not working as expected and it is not triggered as mentioned time in build periodically. please find the attached thread dumps as well. jenkins thread dumps.txt

          Sekhar Babu added a comment - I am also facing the same problem from past 1 week. Jenkins scheduler is not working as expected and it is not triggered as mentioned time in build periodically. please find the attached thread dumps as well.  jenkins thread dumps.txt

          Daniel Beck added a comment -

          sekhar_babu

          same problem

          It's not; yours is stash pull request builder performing expensive networking operations on the thread processing triggers.

          Daniel Beck added a comment - sekhar_babu same problem It's not; yours is stash pull request builder performing expensive networking operations on the thread processing triggers.

          Sekhar Babu added a comment -

          I have installed in last 2 weeks back scheduler worked fine after that as well. 

          Sekhar Babu added a comment - I have installed in last 2 weeks back scheduler worked fine after that as well. 

          Sekhar Babu added a comment -

          After uninstalling the stash pull request  builder plugin scheduler is working fine . what would be the possible way if need both stash pull requet builder and schedulers as well. 

          Sekhar Babu added a comment - After uninstalling the stash pull request  builder plugin scheduler is working fine . what would be the possible way if need both stash pull requet builder and schedulers as well. 

          Jagadish added a comment -

          Even the same issue for me as well. Jenkins core trigger system has been affected due to stash pull request builder plugin :|

          Jagadish added a comment - Even the same issue for me as well. Jenkins core trigger system has been affected due to stash pull request builder plugin :|

          roshan chandekar added a comment -

          We too are facing the same issue after upgrading the Jenkins to 2.73 version.

          Suddenly poll scm and timer builds stop working. Jenkins restart fixed the issue but based  on other jenkins forum looks this issue will come again. Please let us know if permanent solution for this issue.

          roshan chandekar added a comment - We too are facing the same issue after upgrading the Jenkins to 2.73 version. Suddenly poll scm and timer builds stop working. Jenkins restart fixed the issue but based  on other jenkins forum looks this issue will come again. Please let us know if permanent solution for this issue.

          HARISHKUMAR TR added a comment -

          david_resnick Can it be possible to assign this to any of known developers??

          HARISHKUMAR TR added a comment - david_resnick Can it be possible to assign this to any of known developers??

          David Resnick added a comment -

          harishkumartr22 I think someone has to be convinced to take it.

          David Resnick added a comment - harishkumartr22 I think someone has to be convinced to take it.

          Chandan Kumar added a comment -

          This is happening to my Jenkins jobs as well. I wanted to run a job at exactly 8:25 am every day with this cron expression: "25 8 * * *". This was working fine for months until now. Changing the exact minute to 'H' seems to help sometimes. e.g. "H 8 * * *" triggers the job, but not at the exact time I need it to trigger. That's a workaround that might work for me, but it's disappointing to not be able to debug this. Cron log is not much help:

          cron checking hudson.model.FreeStyleProject@505fc44a[Cleanup_Builds/my-job] with spec ‘25 8 * * *’
          Dec 17, 2019 8:25:13 AM FINER hudson.triggers.Trigger
          did not trigger hudson.model.FreeStyleProject@505fc44a[Cleanup_Builds/my-job]
          Dec 17, 2019 8:25:13 AM FINE hudson.triggers.Trigger

           

          Chandan Kumar added a comment - This is happening to my Jenkins jobs as well. I wanted to run a job at exactly 8:25 am every day with this cron expression: "25 8 * * *". This was working fine for months until now. Changing the exact minute to 'H' seems to help sometimes. e.g. "H 8 * * *" triggers the job, but not at the exact time I need it to trigger. That's a workaround that might work for me, but it's disappointing to not be able to debug this. Cron log is not much help: cron checking hudson.model.FreeStyleProject@505fc44a [Cleanup_Builds/my-job] with spec ‘25 8 * * *’ Dec 17, 2019 8:25:13 AM FINER hudson.triggers.Trigger did not trigger hudson.model.FreeStyleProject@505fc44a [Cleanup_Builds/my-job] Dec 17, 2019 8:25:13 AM FINE hudson.triggers.Trigger  

            Unassigned Unassigned
            david_resnick David Resnick
            Votes:
            13 Vote for this issue
            Watchers:
            25 Start watching this issue

              Created:
              Updated: