Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-19445

Jobs randomly stuck with "building remotely on slave-name" message

    XMLWordPrintable

Details

    Description

      Hi folks,

      Last week, I started to experience a weird issue with freezing/endless waiting of building jobs on slave machines. It first started with WinXP 32 bit slave, but I'm now experiencing the same problem also with WinXP 64 bit slave.

      The symptoms are like this...
      The job is started by SCM change or predefined time.
      Both slave machines are correctly started (the last line, in their respective logs, says "Slave successfully connected and online").
      Now the problem is, that the last line in console says "Building remotely on slave-name..." and there is just scrolling the progress circle. Nothing else happens. No GIT or SVN activity, no building, no testing, simply nothing and it stays like that until I kill the job manually.

      Sometimes, the problem happens with just one machine, and the other one runs OK. But today it struck both machines.

      Last week, when the problem started, I tried to update the Jenkins and both slaves with the most recent versions and latest plugins (except the "Git Client Plugin", because of a known issue with building history). Few hours all worked great, then the problem reappeared.

      Please see the attached files with logs and thread dumps from both slave machines.

      What struck me is the high number of threads started on both machines, which looks really weird? But I don't understand Jenkins that deep so I can't say for sure if it's normal or not?

      Any idea what to try or how to solve this behavior?

      Have a nice day,
      Pavel

      Attachments

        1. LB3D-MASTER_05_09_13.txt
          53 kB
        2. LB3D-MASTER.txt
          41 kB
        3. LB3D-Wxpp32sp3_05_09_13.txt
          6 kB
        4. LB3D-Wxpp32sp3.txt
          89 kB
        5. LB3D-Wxpp64sp2_05_09_13.txt
          7 kB
        6. LB3D-Wxpp64sp2.txt
          92 kB
        7. remoting-2.47.patch
          2 kB
        8. threadump.txt
          9 kB

        Issue Links

          Activity

            bbva BBVA added a comment -

            Dave Taddei, can you confirm you are using the property hudson.remoting.RemoteClassLoader.force to force the load of com.sun.jna.Native. Also that you can see that the class is being loaded via the log of the slave.

            bbva BBVA added a comment - Dave Taddei, can you confirm you are using the property hudson.remoting.RemoteClassLoader.force to force the load of com.sun.jna.Native. Also that you can see that the class is being loaded via the log of the slave.
            drtaddei Dave Taddei added a comment - - edited

            Hi BBVA, it does not appear so but I am unsure where to look. Should this be set in the JAVA_ARGS for the master, slave or both? I was also under the impression that had been added to release 2.47 so didn't set anything.

            drtaddei Dave Taddei added a comment - - edited Hi BBVA, it does not appear so but I am unsure where to look. Should this be set in the JAVA_ARGS for the master, slave or both? I was also under the impression that had been added to release 2.47 so didn't set anything.
            bbva BBVA added a comment -

            drtaddei set JVM Options of the slave with: -Dhudson.remoting.RemoteClassLoader.force=com.sun.jna.Native

            bbva BBVA added a comment - drtaddei set JVM Options of the slave with: -Dhudson.remoting.RemoteClassLoader.force=com.sun.jna.Native
            gregcovertsmith Greg Smith added a comment -

            We are using dynamic slaves, with the vcenter plugin. The vcenter plugin does not allow the specifications of any JVM options.

            At least, I do not believe it does. Could the above change be made permanent, or set as the default for all slaves some how?

            gregcovertsmith Greg Smith added a comment - We are using dynamic slaves, with the vcenter plugin. The vcenter plugin does not allow the specifications of any JVM options. At least, I do not believe it does. Could the above change be made permanent, or set as the default for all slaves some how?
            jglick Jesse Glick added a comment -

            gregcovertsmith et al.—see the last comment I left in JENKINS-39179: I wrote a little plugin which might work around JNA deadlocks without needing individual configuration of static agents.

            jglick Jesse Glick added a comment - gregcovertsmith et al.—see the last comment I left in  JENKINS-39179 : I wrote a little plugin which might work around JNA deadlocks without needing individual configuration of static agents.

            People

              schristou Steven Christou
              odklizec Pavel Kudrys
              Votes:
              14 Vote for this issue
              Watchers:
              26 Start watching this issue

              Dates

                Created:
                Updated: