Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-14752

SCM Polling / Max # of concurrent polling = 1 hangs github polling

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • core
    • None
    • ubuntu x32, java 7x32, jenkins 1.476, git 1.7.9.5

      SCM Polling / Max # of concurrent polling = 1 hangs github polling

      after a while process list shows a lot of entries like this, for each project in jenkins

      jenkins 18383 0.0 0.0 6208 1192 ? S 17:09 0:00 git fetch -t git@github.com:carrot-garden/carrot-util.git +refs/heads/:refs/remotes/origin/
      jenkins 18388 0.0 0.0 6300 2332 ? S 17:09 0:00 ssh git@github.com git-upload-pack 'carrot-garden/carrot-util.git'

      which appear to be in hang state;

      github scm polling works no more;

      resolution - periodic
      "killall git ssh"

          [JENKINS-14752] SCM Polling / Max # of concurrent polling = 1 hangs github polling

          Andrei Pozolotin created issue -
          Andrei Pozolotin made changes -
          Description New:
          SCM Polling / Max # of concurrent polling = 1 hangs github polling

          after a while process list shows a lot of entries like this, for each project in jenkins

          jenkins 18383 0.0 0.0 6208 1192 ? S 17:09 0:00 git fetch -t git@github.com:carrot-garden/carrot-util.git +refs/heads/*:refs/remotes/origin/*
          jenkins 18388 0.0 0.0 6300 2332 ? S 17:09 0:00 ssh git@github.com git-upload-pack 'carrot-garden/carrot-util.git'

          which appear to be in hang state;

          github scm polling works no more;

          resolution - periodic
          "killall git ssh"
          Environment Original: ubuntu x32, java 3x32, jenkins 1.476

          SCM Polling / Max # of concurrent polling = 1 hangs github polling

          after a while process list shows a lot of entries like this, for each project in jenkins

          jenkins 18383 0.0 0.0 6208 1192 ? S 17:09 0:00 git fetch -t git@github.com:carrot-garden/carrot-util.git +refs/heads/*:refs/remotes/origin/*
          jenkins 18388 0.0 0.0 6300 2332 ? S 17:09 0:00 ssh git@github.com git-upload-pack 'carrot-garden/carrot-util.git'

          which appear to be in hang state;

          github scm polling works no more;

          resolution - periodic
          "killall git ssh"
          New: ubuntu x32, java 3x32, jenkins 1.476
          Andrei Pozolotin made changes -
          Description Original:
          SCM Polling / Max # of concurrent polling = 1 hangs github polling

          after a while process list shows a lot of entries like this, for each project in jenkins

          jenkins 18383 0.0 0.0 6208 1192 ? S 17:09 0:00 git fetch -t git@github.com:carrot-garden/carrot-util.git +refs/heads/*:refs/remotes/origin/*
          jenkins 18388 0.0 0.0 6300 2332 ? S 17:09 0:00 ssh git@github.com git-upload-pack 'carrot-garden/carrot-util.git'

          which appear to be in hang state;

          github scm polling works no more;

          resolution - periodic
          "killall git ssh"
          New: SCM Polling / Max # of concurrent polling = 1 hangs github polling

          after a while process list shows a lot of entries like this, for each project in jenkins

          jenkins 18383 0.0 0.0 6208 1192 ? S 17:09 0:00 git fetch -t git@github.com:carrot-garden/carrot-util.git +refs/heads/*:refs/remotes/origin/*
          jenkins 18388 0.0 0.0 6300 2332 ? S 17:09 0:00 ssh git@github.com git-upload-pack 'carrot-garden/carrot-util.git'

          which appear to be in hang state;

          github scm polling works no more;

          resolution - periodic
          "killall git ssh"
          Environment Original: ubuntu x32, java 3x32, jenkins 1.476
          New: ubuntu x32, java 7x32, jenkins 1.476
          Andrei Pozolotin made changes -
          Environment Original: ubuntu x32, java 7x32, jenkins 1.476
          New: ubuntu x32, java 7x32, jenkins 1.476, git 1.7.9.5

          Same here with the following env:

          • git version: 1.7.2.5
          • Debian: 6.0.5
          • Java: Sun/Oracle JDK 1.6.0_26

          Sascha Vogt added a comment - Same here with the following env: git version: 1.7.2.5 Debian: 6.0.5 Java: Sun/Oracle JDK 1.6.0_26

          Daniel Beck added a comment -

          This option exists to restrict parallel SCM operations when you have several dozens, hundreds or thousands of jobs to a limit of maybe 10-50. What's the rationale to enter "1"?

          Daniel Beck added a comment - This option exists to restrict parallel SCM operations when you have several dozens, hundreds or thousands of jobs to a limit of maybe 10-50. What's the rationale to enter "1"?

          Well, it doesn't matter if you have 1 or 10 concurrent. Sometimes the git command just hangs. Having 10 concurrent pollings just means it needs to happen 10 times before you're completely blocked out. Even if you don't restrict, you will at one point have hundreds or thousands of hanging pollings.

          I think the right thing here would be to configure a timeout and if that is reached kill the process.

          Sascha Vogt added a comment - Well, it doesn't matter if you have 1 or 10 concurrent. Sometimes the git command just hangs. Having 10 concurrent pollings just means it needs to happen 10 times before you're completely blocked out. Even if you don't restrict, you will at one point have hundreds or thousands of hanging pollings. I think the right thing here would be to configure a timeout and if that is reached kill the process.

          Mig Jacq added a comment -

          This is suddenly happening to me too on LTS release 1.554.2 and Ubuntu 12.04 LTS, git version 1.7.9.5. I have verified that it's not a network issue as github is reachable, it seems sometimes the process just hangs with read() (when strace'ing it).

          Mig Jacq added a comment - This is suddenly happening to me too on LTS release 1.554.2 and Ubuntu 12.04 LTS, git version 1.7.9.5. I have verified that it's not a network issue as github is reachable, it seems sometimes the process just hangs with read() (when strace'ing it).

          Rob Duff added a comment -

          Something similar is happening with me, although it has nothing to do with the "max # of concurrent polling" setting in my case. The ssh process gets hung up when it runs "git-upload-pack". It seems to happen when we're grabbing the same repo on another executor on the same machine at the exact same time. It makes me think that this is the same issue as what you're seeing, but manifested by different means.

          Another problem we get when we grab the same repo on the same machine at the same time is https://issues.jenkins-ci.org/browse/JENKINS-24179. I won't go so far as to say that they are the same issue, but it has some potential to be somehow related. If that is the case, then it might be a git/ssh concurrency issue and not so much of a Jenkins issue. But – I can't reproduce this outside of Jenkins yet, so... I can't really say for sure.

          I'd be interested to know if the other cases happen to have the same repo accessed at the same time on the same machine.

          Rob Duff added a comment - Something similar is happening with me, although it has nothing to do with the "max # of concurrent polling" setting in my case. The ssh process gets hung up when it runs "git-upload-pack". It seems to happen when we're grabbing the same repo on another executor on the same machine at the exact same time. It makes me think that this is the same issue as what you're seeing, but manifested by different means. Another problem we get when we grab the same repo on the same machine at the same time is https://issues.jenkins-ci.org/browse/JENKINS-24179 . I won't go so far as to say that they are the same issue, but it has some potential to be somehow related. If that is the case, then it might be a git/ssh concurrency issue and not so much of a Jenkins issue. But – I can't reproduce this outside of Jenkins yet, so... I can't really say for sure. I'd be interested to know if the other cases happen to have the same repo accessed at the same time on the same machine.

          David Feldsine added a comment - - edited

          I am having the same issue.
          git 1.7.1
          Jenkins 1.492
          Git Plugin 1.4.0
          RedHat 2.6.32-279.5.2.el6.x86_64

          I will go for weeks with no issue then I will have a flurry of them. I would say that it fails about .1% to .01% of the time.

          jenkins  18718  0.0  0.0  10136  1240 ?        S    16:59   0:00 git fetch -t origin +refs/heads/*:refs/remotes/origin/*
          jenkins  18722  0.0  0.0  60040  3048 ?        S    16:59   0:00 ssh git@bitbucket.org git-upload-pack 'company/REPO.git'
          jenkins  29927  0.0  0.0  37336  3640 ?        S    16:01   0:00 git fetch -t origin +refs/heads/*:refs/remotes/origin/*
          jenkins  29931  0.0  0.0  60040  3048 ?        S    16:01   0:00 ssh git@bitbucket.org git-upload-pack 'company/REPO2.git'
          

          David Feldsine added a comment - - edited I am having the same issue. git 1.7.1 Jenkins 1.492 Git Plugin 1.4.0 RedHat 2.6.32-279.5.2.el6.x86_64 I will go for weeks with no issue then I will have a flurry of them. I would say that it fails about .1% to .01% of the time. jenkins 18718 0.0 0.0 10136 1240 ? S 16:59 0:00 git fetch -t origin +refs/heads/*:refs/remotes/origin/* jenkins 18722 0.0 0.0 60040 3048 ? S 16:59 0:00 ssh git@bitbucket.org git-upload-pack 'company/REPO.git' jenkins 29927 0.0 0.0 37336 3640 ? S 16:01 0:00 git fetch -t origin +refs/heads/*:refs/remotes/origin/* jenkins 29931 0.0 0.0 60040 3048 ? S 16:01 0:00 ssh git@bitbucket.org git-upload-pack 'company/REPO2.git'

            Unassigned Unassigned
            andrei_pozolotin Andrei Pozolotin
            Votes:
            2 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated: