-
Bug
-
Resolution: Won't Fix
-
Critical
-
FreeBSD 10.1
Jenkins 1.598
git 2.3.5
git-client 1.16.1
scm-api 0.2
Our cron-based polling does not work. The UI correctly seems to say when the next and previous polling times will/would be, but the polling log isn't updated. Sometimes it's more than 3 days behind.
I found this in the threadDump page:
Jenkins cron thread "Jenkins cron thread" Id=25 Group=main WAITING on java.util.TaskQueue@1c75ef13 at java.lang.Object.wait(Native Method) - waiting on java.util.TaskQueue@1c75ef13 at java.lang.Object.wait(Object.java:503) at java.util.TimerThread.mainLoop(Timer.java:526) at java.util.TimerThread.run(Timer.java:505) jenkins.util.Timer [#10] "jenkins.util.Timer [#10]" Id=78 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f92c0bf at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f92c0bf at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) jenkins.util.Timer [#1] "jenkins.util.Timer [#1]" Id=26 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f92c0bf at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f92c0bf at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) jenkins.util.Timer [#2] "jenkins.util.Timer [#2]" Id=63 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f92c0bf at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f92c0bf at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) jenkins.util.Timer [#3] "jenkins.util.Timer [#3]" Id=66 Group=main TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f92c0bf at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f92c0bf at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)
[JENKINS-27677] SCM Polling does not run
Frank van Gemeren
created issue -
Frank van Gemeren
added a comment - It ran twice on a Saturday while that's not possible according to one of the crons:
H/30 7-22 * * 1-5
The server time is correct.
This is the whole System Information page: https://gist.github.com/frvge/28884e6755a2610ec2cb (removed a few IPs).
We upgraded to the version with the new build layout so I'd prefer not to downgrade. With > 500 jobs I don't want to take chances that something goes wrong. I can do it, but only as a last resort.
Jenkins is running in a screen session. Maybe that also has something to do with it?
Frank van Gemeren
added a comment - It ran twice on a Saturday while that's not possible according to one of the crons:
H/30 7-22 * * 1-5
The server time is correct.
This is the whole System Information page: https://gist.github.com/frvge/28884e6755a2610ec2cb (removed a few IPs).
We upgraded to the version with the new build layout so I'd prefer not to downgrade. With > 500 jobs I don't want to take chances that something goes wrong. I can do it, but only as a last resort.
Jenkins is running in a screen session. Maybe that also has something to do with it?
Frank van Gemeren
added a comment - Update:
We just did maintenance on our gitlab server, including a start/stop of the program, and suddenly all the polling / triggers started working again. This meant a big big big queue. (which failed because we couldn't checkout the code of course).
The interesting part is that a timed job, configured like
58 23 * * *
also ran again automatically. Its running time is also not consistent with that schedule:
Failed > Console Output#16 Apr 1, 2015 6:21 PM Failed > Console Output#15 Apr 1, 2015 6:05 PM Failed > Console Output#14 Apr 1, 2015 6:04 PM Success > Console Output#13 Apr 1, 2015 7:38 AM Success > Console Output#12 Mar 30, 2015 10:24 PM Success > Console Output#11 Mar 29, 2015 11:22 AM Success > Console Output#10 Mar 27, 2015 9:43 PM Success > Console Output#9 Mar 26, 2015 11:18 AM Success > Console Output#8 Mar 25, 2015 4:22 AM Success > Console Output#7 Mar 23, 2015 7:16 PM Success > Console Output#6 Mar 22, 2015 10:23 AM Success > Console Output#5 Mar 21, 2015 1:57 AM Success > Console Output#4 Mar 20, 2015 12:59 PM Success > Console Output#3 Mar 19, 2015 2:40 AM Success > Console Output#2 Mar 17, 2015 8:45 PM Success > Console Output#1 Mar 16, 2015 6:05 PM
Frank van Gemeren
added a comment - Update:
We just did maintenance on our gitlab server, including a start/stop of the program, and suddenly all the polling / triggers started working again. This meant a big big big queue. (which failed because we couldn't checkout the code of course).
The interesting part is that a timed job, configured like
58 23 * * *
also ran again automatically. Its running time is also not consistent with that schedule:
Failed > Console Output#16 Apr 1, 2015 6:21 PM
Failed > Console Output#15 Apr 1, 2015 6:05 PM
Failed > Console Output#14 Apr 1, 2015 6:04 PM
Success > Console Output#13 Apr 1, 2015 7:38 AM
Success > Console Output#12 Mar 30, 2015 10:24 PM
Success > Console Output#11 Mar 29, 2015 11:22 AM
Success > Console Output#10 Mar 27, 2015 9:43 PM
Success > Console Output#9 Mar 26, 2015 11:18 AM
Success > Console Output#8 Mar 25, 2015 4:22 AM
Success > Console Output#7 Mar 23, 2015 7:16 PM
Success > Console Output#6 Mar 22, 2015 10:23 AM
Success > Console Output#5 Mar 21, 2015 1:57 AM
Success > Console Output#4 Mar 20, 2015 12:59 PM
Success > Console Output#3 Mar 19, 2015 2:40 AM
Success > Console Output#2 Mar 17, 2015 8:45 PM
Success > Console Output#1 Mar 16, 2015 6:05 PM
I wonder if that hints the polling has started but not completed? The calls to command line git from the git plugin attempt to never have the command line git block, but there seem to be more ways to block than there are techniques to prevent blocking.
You could check the polling log of the jobs to see if the polling is in progress. Alternately, you could check the processes on the master server to see if there are many git processes running which have existed for a long time.
Frank van Gemeren
added a comment - There are a few long running git processes. Example:
jenkins 74321 0.0 0.0 52568 4484 1 I+ 16Mar15 0:00.03 ssh git@git.company.com git-upload-pack 'repoowner/reponame.git'
Frank van Gemeren
added a comment - There are a few long running git processes. Example:
jenkins 74321 0.0 0.0 52568 4484 1 I+ 16Mar15 0:00.03 ssh git@git.company.com git-upload-pack 'repoowner/reponame.git'
Frank van Gemeren
added a comment - I've updated to 1.607 and we've changed some crons to a simpler
*/15 * * *
I'll let you know.
Frank van Gemeren
added a comment - I've updated to 1.607 and we've changed some crons to a simpler
*/15 * * *
I'll let you know. Frank van Gemeren
added a comment - It works on some projects now, but it's still way below what we expect. Our main project with a lot of commits hasn't run since last Friday.
Might be related to: https://issues.jenkins-ci.org/browse/JENKINS-26208 . I'll upgrade to 1.608.
Frank van Gemeren
added a comment - It works on some projects now, but it's still way below what we expect. Our main project with a lot of commits hasn't run since last Friday.
Might be related to: https://issues.jenkins-ci.org/browse/JENKINS-26208 . I'll upgrade to 1.608. Frank van Gemeren
added a comment - Is there a way to check the output of "isClogged()" in a log? Like
Frank van Gemeren
added a comment - Is there a way to check the output of "isClogged()" in a log? Like
https://github.com/jenkinsci/jenkins/blob/608517e187cb5bd1566b1c3728a4df0f7ac4dd5c/core/src/main/java/hudson/triggers/SCMTrigger.java#L225 Frank van Gemeren
added a comment - Just saw this in the log. This was in a merge request. We use the Gitlab Merge Request Builder plugin for that.
Apr 07, 2015 3:59:42 PM hudson.triggers.Trigger checkTriggers WARNING: org.jenkinsci.plugins.gitlab.GitlabBuildTrigger.run() failed for hudson.model.FreeStyleProject@225cf060[myproject] java.lang.NullPointerException at org.jenkinsci.plugins.gitlab.GitlabBuildTrigger.run(GitlabBuildTrigger.java:100) at hudson.triggers.Trigger.checkTriggers(Trigger.java:265) at hudson.triggers.Trigger$Cron.doRun(Trigger.java:214) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)
Frank van Gemeren
added a comment - Just saw this in the log. This was in a merge request. We use the Gitlab Merge Request Builder plugin for that.
Apr 07, 2015 3:59:42 PM hudson.triggers.Trigger checkTriggers
WARNING: org.jenkinsci.plugins.gitlab.GitlabBuildTrigger.run() failed for hudson.model.FreeStyleProject@225cf060[myproject]
java.lang.NullPointerException
at org.jenkinsci.plugins.gitlab.GitlabBuildTrigger.run(GitlabBuildTrigger.java:100)
at hudson.triggers.Trigger.checkTriggers(Trigger.java:265)
at hudson.triggers.Trigger$Cron.doRun(Trigger.java:214)
at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang. Thread .run( Thread .java:745)
Can you provide more details to describe conditions which may be causing the problem?
Does the problem persist if you switch to the long term support Jenkins version (currently 1.596.3)?
Does the problem persist on a new installation of the latest Jenkins version?
What is the JDK version running on your FreeBSD system?