Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-19445

Jobs randomly stuck with "building remotely on slave-name" message

      Hi folks,

      Last week, I started to experience a weird issue with freezing/endless waiting of building jobs on slave machines. It first started with WinXP 32 bit slave, but I'm now experiencing the same problem also with WinXP 64 bit slave.

      The symptoms are like this...
      The job is started by SCM change or predefined time.
      Both slave machines are correctly started (the last line, in their respective logs, says "Slave successfully connected and online").
      Now the problem is, that the last line in console says "Building remotely on slave-name..." and there is just scrolling the progress circle. Nothing else happens. No GIT or SVN activity, no building, no testing, simply nothing and it stays like that until I kill the job manually.

      Sometimes, the problem happens with just one machine, and the other one runs OK. But today it struck both machines.

      Last week, when the problem started, I tried to update the Jenkins and both slaves with the most recent versions and latest plugins (except the "Git Client Plugin", because of a known issue with building history). Few hours all worked great, then the problem reappeared.

      Please see the attached files with logs and thread dumps from both slave machines.

      What struck me is the high number of threads started on both machines, which looks really weird? But I don't understand Jenkins that deep so I can't say for sure if it's normal or not?

      Any idea what to try or how to solve this behavior?

      Have a nice day,
      Pavel

        1. LB3D-MASTER_05_09_13.txt
          53 kB
        2. LB3D-MASTER.txt
          41 kB
        3. LB3D-Wxpp32sp3_05_09_13.txt
          6 kB
        4. LB3D-Wxpp32sp3.txt
          89 kB
        5. LB3D-Wxpp64sp2_05_09_13.txt
          7 kB
        6. LB3D-Wxpp64sp2.txt
          92 kB
        7. remoting-2.47.patch
          2 kB
        8. threadump.txt
          9 kB

          [JENKINS-19445] Jobs randomly stuck with "building remotely on slave-name" message

          Oleg Nenashev added a comment -

          ddaumiller
          Jenkins 2 LTS - yes
          1.651.3 - under review: https://github.com/jenkinsci/jenkins/pull/2357

          Oleg Nenashev added a comment - ddaumiller Jenkins 2 LTS - yes 1.651.3 - under review: https://github.com/jenkinsci/jenkins/pull/2357

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          pom.xml
          http://jenkins-ci.org/commit/jenkins/12e79963cca5122351943ee107f65c3ad91a2e25
          Log:
          [JENKINS-19445, JENKINS-34213, JENKINS-34808, JENKINS-34121] Bump remoting to 2.59. (#2344)

          • [JENKINS-19445, JENKINS-34213, JENKINS-34808] Bump remoting to 2.58.

          Changes:

          (cherry picked from commit 409438f36dc80f20964fb16f8d88041e11ba4ed4)

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: pom.xml http://jenkins-ci.org/commit/jenkins/12e79963cca5122351943ee107f65c3ad91a2e25 Log: [JENKINS-19445, JENKINS-34213, JENKINS-34808, JENKINS-34121] Bump remoting to 2.59. (#2344) [JENKINS-19445, JENKINS-34213, JENKINS-34808] Bump remoting to 2.58. Changes: JENKINS-34213 ( https://issues.jenkins-ci.org/browse/JENKINS-34213 ) - Ensure that the unexporter cleans up whatever it can each sweep ( https://github.com/jenkinsci/remoting/pull/81 ) JENKINS-19445 ( https://issues.jenkins-ci.org/browse/JENKINS-19445 ) Force class load on UserRequest in order to prevent deadlock on windows nodes when using JNA and Subversion ( https://github.com/jenkinsci/remoting/pull/81 ) JENKINS-34808 ( https://issues.jenkins-ci.org/browse/JENKINS-34808 ) - Allow user to adjust socket timeout ( https://github.com/jenkinsci/remoting/pull/68 ) JENKINS-34121 - Upgrade remoting to 2.59 (cherry picked from commit 409438f36dc80f20964fb16f8d88041e11ba4ed4)

          Oleg Nenashev added a comment -

          Released as Jenkins-2.4

          Oleg Nenashev added a comment - Released as Jenkins-2.4

          Dave Taddei added a comment - - edited

          We are showing this happening on the latest jenkins build (2.17) and latest slave jar (2.60) on a Windows 7 64-bit slave. Appears to be an intermittent fault but almost always occurs when an on-demand build is taking place.

          Dave Taddei added a comment - - edited We are showing this happening on the latest jenkins build (2.17) and latest slave jar (2.60) on a Windows 7 64-bit slave. Appears to be an intermittent fault but almost always occurs when an on-demand build is taking place.

          Daniel Beck added a comment -

          drtaddei

          latest jenkins build … latest slave jar

          Please be more specific. What version?

          Daniel Beck added a comment - drtaddei latest jenkins build … latest slave jar Please be more specific. What version?

          BBVA added a comment -

          Dave Taddei, can you confirm you are using the property hudson.remoting.RemoteClassLoader.force to force the load of com.sun.jna.Native. Also that you can see that the class is being loaded via the log of the slave.

          BBVA added a comment - Dave Taddei, can you confirm you are using the property hudson.remoting.RemoteClassLoader.force to force the load of com.sun.jna.Native. Also that you can see that the class is being loaded via the log of the slave.

          Dave Taddei added a comment - - edited

          Hi BBVA, it does not appear so but I am unsure where to look. Should this be set in the JAVA_ARGS for the master, slave or both? I was also under the impression that had been added to release 2.47 so didn't set anything.

          Dave Taddei added a comment - - edited Hi BBVA, it does not appear so but I am unsure where to look. Should this be set in the JAVA_ARGS for the master, slave or both? I was also under the impression that had been added to release 2.47 so didn't set anything.

          BBVA added a comment -

          drtaddei set JVM Options of the slave with: -Dhudson.remoting.RemoteClassLoader.force=com.sun.jna.Native

          BBVA added a comment - drtaddei set JVM Options of the slave with: -Dhudson.remoting.RemoteClassLoader.force=com.sun.jna.Native

          Greg Smith added a comment -

          We are using dynamic slaves, with the vcenter plugin. The vcenter plugin does not allow the specifications of any JVM options.

          At least, I do not believe it does. Could the above change be made permanent, or set as the default for all slaves some how?

          Greg Smith added a comment - We are using dynamic slaves, with the vcenter plugin. The vcenter plugin does not allow the specifications of any JVM options. At least, I do not believe it does. Could the above change be made permanent, or set as the default for all slaves some how?

          Jesse Glick added a comment -

          gregcovertsmith et al.—see the last comment I left in JENKINS-39179: I wrote a little plugin which might work around JNA deadlocks without needing individual configuration of static agents.

          Jesse Glick added a comment - gregcovertsmith et al.—see the last comment I left in  JENKINS-39179 : I wrote a little plugin which might work around JNA deadlocks without needing individual configuration of static agents.

            schristou Steven Christou
            odklizec Pavel Kudrys
            Votes:
            14 Vote for this issue
            Watchers:
            26 Start watching this issue

              Created:
              Updated: