• Icon: Bug Bug
    • Resolution: Won't Fix
    • Icon: Critical Critical
    • git-plugin
    • FreeBSD 10.1
      Jenkins 1.598
      git 2.3.5
      git-client 1.16.1
      scm-api 0.2

      Our cron-based polling does not work. The UI correctly seems to say when the next and previous polling times will/would be, but the polling log isn't updated. Sometimes it's more than 3 days behind.

      I found this in the threadDump page:

      Jenkins cron thread
      
      "Jenkins cron thread" Id=25 Group=main WAITING on java.util.TaskQueue@1c75ef13
      	at java.lang.Object.wait(Native Method)
      	-  waiting on java.util.TaskQueue@1c75ef13
      	at java.lang.Object.wait(Object.java:503)
      	at java.util.TimerThread.mainLoop(Timer.java:526)
      	at java.util.TimerThread.run(Timer.java:505)
      
      jenkins.util.Timer [#10]
      
      "jenkins.util.Timer [#10]" Id=78 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f92c0bf
      	at sun.misc.Unsafe.park(Native Method)
      	-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f92c0bf
      	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
      	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
      
      jenkins.util.Timer [#1]
      
      "jenkins.util.Timer [#1]" Id=26 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f92c0bf
      	at sun.misc.Unsafe.park(Native Method)
      	-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f92c0bf
      	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
      	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
      
      jenkins.util.Timer [#2]
      
      "jenkins.util.Timer [#2]" Id=63 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f92c0bf
      	at sun.misc.Unsafe.park(Native Method)
      	-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f92c0bf
      	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
      	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
      
      jenkins.util.Timer [#3]
      
      "jenkins.util.Timer [#3]" Id=66 Group=main TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f92c0bf
      	at sun.misc.Unsafe.park(Native Method)
      	-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f92c0bf
      	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
      	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
      

          [JENKINS-27677] SCM Polling does not run

          Just saw this in the log. This was in a merge request. We use the Gitlab Merge Request Builder plugin for that.

          Apr 07, 2015 3:59:42 PM hudson.triggers.Trigger checkTriggers
          WARNING: org.jenkinsci.plugins.gitlab.GitlabBuildTrigger.run() failed for hudson.model.FreeStyleProject@225cf060[myproject]
          java.lang.NullPointerException
                  at org.jenkinsci.plugins.gitlab.GitlabBuildTrigger.run(GitlabBuildTrigger.java:100)
                  at hudson.triggers.Trigger.checkTriggers(Trigger.java:265)
                  at hudson.triggers.Trigger$Cron.doRun(Trigger.java:214)
                  at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51)
                  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
                  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
                  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
                  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
                  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
                  at java.lang.Thread.run(Thread.java:745)
          

          Frank van Gemeren added a comment - Just saw this in the log. This was in a merge request. We use the Gitlab Merge Request Builder plugin for that. Apr 07, 2015 3:59:42 PM hudson.triggers.Trigger checkTriggers WARNING: org.jenkinsci.plugins.gitlab.GitlabBuildTrigger.run() failed for hudson.model.FreeStyleProject@225cf060[myproject] java.lang.NullPointerException at org.jenkinsci.plugins.gitlab.GitlabBuildTrigger.run(GitlabBuildTrigger.java:100) at hudson.triggers.Trigger.checkTriggers(Trigger.java:265) at hudson.triggers.Trigger$Cron.doRun(Trigger.java:214) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang. Thread .run( Thread .java:745)

          We're now at 1.609 . Our slaves are also updated to the latest slave.jar . Issue persists. Would like to be able to see if the polling thread is clogged up or not.

          Frank van Gemeren added a comment - We're now at 1.609 . Our slaves are also updated to the latest slave.jar . Issue persists. Would like to be able to see if the polling thread is clogged up or not.

          Is there an update on this?

          We're upgrading our Jenkins-Job-Builder program soon. Some issues that were fixed had to do with polling. Even if they caused it, which isn't certain until we've tested it, I'd like to see a list of the "clogged" polls in Jenkins.

          Frank van Gemeren added a comment - Is there an update on this? We're upgrading our Jenkins-Job-Builder program soon. Some issues that were fixed had to do with polling. Even if they caused it, which isn't certain until we've tested it, I'd like to see a list of the "clogged" polls in Jenkins.

          Mark Waite added a comment -

          I am not doing any investigation or work on this question.

          Mark Waite added a comment - I am not doing any investigation or work on this question.

          What can I do to change that? It's kinda annoying when we're moving to continuous integration and only a handful of our jobs that poll work (and even then not as often as they should).

          Frank van Gemeren added a comment - What can I do to change that? It's kinda annoying when we're moving to continuous integration and only a handful of our jobs that poll work (and even then not as often as they should).

          Mark Waite added a comment -

          Unfortunately, I don't think there is much you can do to change that (at least for me). Sorry about that. I think the investigation work will need to come from you.

          I can't duplicate your problem.

          I don't have a FreeBSD system in my git plugin / git client plugin test environment (only Windows 7 x86, Windows 7 x64, Windows 8.1 x64, Windows Home Server 2011 x64, Debian 6 x86, Debian 7 x64, Debian 8 x64, Ubuntu 14.04 x64, CentOS 6 x86, and CentOS 7 x64).

          I don't have a GitLab server in my test environment (only GitHub, gitweb, bitbucket, sourceforge, and a few others).

          I'm a volunteer who contributes to the plugins on my personal time, so you'd need to persuade me to use my personal time to work on a problem which I can't duplicate, and which has hints that it may be related to a local configuration problem. Some of the things I find most persuasive are things like:

          • Expand the understanding of the platforms where the bug happens and does not happen, so that its impact to the larger community is more clear. If, for example, the problem also appears when using GitHub or bitbucket or a git protocol server on CentOS or Debian, that makes it more interesting to me
          • Expand the understanding of the versions where the bug happens and does not happen, so that its impact to the larger community is more clear. If, for example, the problem also appears when using the most recent long term support release, that is more interesting to me, since I'm a user of the long term support release
          • Expand the understanding of the conditions where the problem happens and does not happen, so that its impact to the larger community is more clear. If, for example, the problem does not appear when using git plugin 2.3.4 and does appear in 2.3.5, then that is very interesting, since it hints that there was a plugin version which behaved more the way you wanted

          If you'd rather have a more direct form of persuasion, you might be able to persuade others to investigate the problem by paying them to investigate. There is also freedomsponsors.org which provides a way to offer to fund a bug fix or investigation. I don't think my employer will allow me to take money for my personal development activities, so neither of those techniques will persuade me. It's not a personal thing, nor is it me saying that the problem you've found is not a bug.

          I hope that my description is not viewed as an attempt to offend or rebuff your concern. It is not. I'm trying to help you see which things might persuade others to help, and which will not.

          Mark Waite added a comment - Unfortunately, I don't think there is much you can do to change that (at least for me). Sorry about that. I think the investigation work will need to come from you. I can't duplicate your problem. I don't have a FreeBSD system in my git plugin / git client plugin test environment (only Windows 7 x86, Windows 7 x64, Windows 8.1 x64, Windows Home Server 2011 x64, Debian 6 x86, Debian 7 x64, Debian 8 x64, Ubuntu 14.04 x64, CentOS 6 x86, and CentOS 7 x64). I don't have a GitLab server in my test environment (only GitHub, gitweb, bitbucket, sourceforge, and a few others). I'm a volunteer who contributes to the plugins on my personal time, so you'd need to persuade me to use my personal time to work on a problem which I can't duplicate, and which has hints that it may be related to a local configuration problem. Some of the things I find most persuasive are things like: Expand the understanding of the platforms where the bug happens and does not happen, so that its impact to the larger community is more clear. If, for example, the problem also appears when using GitHub or bitbucket or a git protocol server on CentOS or Debian, that makes it more interesting to me Expand the understanding of the versions where the bug happens and does not happen, so that its impact to the larger community is more clear. If, for example, the problem also appears when using the most recent long term support release, that is more interesting to me, since I'm a user of the long term support release Expand the understanding of the conditions where the problem happens and does not happen, so that its impact to the larger community is more clear. If, for example, the problem does not appear when using git plugin 2.3.4 and does appear in 2.3.5, then that is very interesting, since it hints that there was a plugin version which behaved more the way you wanted If you'd rather have a more direct form of persuasion, you might be able to persuade others to investigate the problem by paying them to investigate. There is also freedomsponsors.org which provides a way to offer to fund a bug fix or investigation. I don't think my employer will allow me to take money for my personal development activities, so neither of those techniques will persuade me. It's not a personal thing, nor is it me saying that the problem you've found is not a bug. I hope that my description is not viewed as an attempt to offend or rebuff your concern. It is not. I'm trying to help you see which things might persuade others to help, and which will not.

          Hi Mark,

          Thank you for your answer. I understand your position. I'll see if I can get approval for using the freedomsponsors option. We also have an in-house Java team so maybe I can use them to debug this further.

          In the meantime I have another, related, question: I just noticed that there's a hudson.triggers.SCMTrigger.starvationThreshold "hidden" feature with its default set to 1 hour (description is "Milliseconds waiting for polling executor before trigger reports it is clogged"). We do a lot of polling, so would extending it to 2 hours or more potentially fix this? If that's the case, maybe it makes sense to use a separate polling thread in the Jenkins core?

          Frank van Gemeren added a comment - Hi Mark, Thank you for your answer. I understand your position. I'll see if I can get approval for using the freedomsponsors option. We also have an in-house Java team so maybe I can use them to debug this further. In the meantime I have another, related, question: I just noticed that there's a hudson.triggers.SCMTrigger.starvationThreshold "hidden" feature with its default set to 1 hour (description is "Milliseconds waiting for polling executor before trigger reports it is clogged"). We do a lot of polling, so would extending it to 2 hours or more potentially fix this? If that's the case, maybe it makes sense to use a separate polling thread in the Jenkins core?

          Mark Waite added a comment -

          frvge I don't know the code related to starvation threshold, so I can't help with that at all. If polling frequency is the concern, you may want to read the "polling must die" blog posting from Kohsuke Kawaguchi. He describes a technique which reduces polling dramatically and also reduces the time between a change being submitted and related jobs starting.

          Mark Waite added a comment - frvge I don't know the code related to starvation threshold, so I can't help with that at all. If polling frequency is the concern, you may want to read the " polling must die " blog posting from Kohsuke Kawaguchi. He describes a technique which reduces polling dramatically and also reduces the time between a change being submitted and related jobs starting.

          Hi Mark, we currently prefer not to run via webhooks, because our tests (and deployments) take 20 minutes, and we get a few pushes per minute. In other words: we didn't want to overload our slaves (16 executors in total).

          We will re-evaluate a web-hook like system and also look into Gerrit and Zuul to limit the runtime.

          Thanks for your time. The ticket can be suspended.

          Frank van Gemeren added a comment - Hi Mark, we currently prefer not to run via webhooks, because our tests (and deployments) take 20 minutes, and we get a few pushes per minute. In other words: we didn't want to overload our slaves (16 executors in total). We will re-evaluate a web-hook like system and also look into Gerrit and Zuul to limit the runtime. Thanks for your time. The ticket can be suspended.

          Mark Waite added a comment -

          Don't plan to fix this report. I can't duplicate it.

          Mark Waite added a comment - Don't plan to fix this report. I can't duplicate it.

            ndeloof Nicolas De Loof
            frvge Frank van Gemeren
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: