Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-35432

Scheduled builds crash after running for more than 8 hours

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      I am using VS online, TFS plugin 4.1.

      I have scheduled builds (every 5 minutes) which checkcode code using the TFS plugin. After 1 or 2 days the builds start failing and I need to restart jenkins.

      From my initial searches, I believe this is an issue with how the plugin interacts with the TFS Java sdk.

      https://social.msdn.microsoft.com/Forums/vstudio/en-US/2ace3851-8e1c-4aee-8e2d-192b0810213a/tfs-sdk-leak?forum=tee

      The workaround is to close the TFSConfigurationServer first and then close the TFSTeamProjectCollection. Future versions of the TFS SDK will take care of closing the TFSConfigurationServer AND throw an exception if you close them the other way around.

      Here is the log:

      FATAL: null
      java.lang.StackOverflowError
      at com.microsoft.tfs.util.listeners.Comparators$IdentityComparator.compare(Comparators.java:57)
      at com.microsoft.tfs.util.listeners.StandardListenerList$ListenerNode.addListener(StandardListenerList.java:330)
      at com.microsoft.tfs.util.listeners.StandardListenerList$ListenerNode.addListener(StandardListenerList.java:343)
      at com.microsoft.tfs.util.listeners.StandardListenerList$ListenerNode.addListener(StandardListenerList.java:343)
      at com.microsoft.tfs.util.listeners.StandardListenerList$ListenerNode.addListener(StandardListenerList.java:343)
      at com.microsoft.tfs.util.listeners.StandardListenerList$ListenerNode.addListener(StandardListenerList.java:343)

        Attachments

          Activity

          Hide
          peaz Mikael Norée added a comment -

          I have the same problem using Jenkins 2.20 and TFS-plugin 5.1.0. After a 2-3 days all jobs start failing with same message in output. Fixed by restarting Jenkins service on the windows server.

          15:31:06 FATAL: null
          15:31:06
          java.lang.StackOverflowError
          15:31:06 at com.microsoft.tfs.util.listeners.StandardListenerList$ListenerNode.addListener(StandardListenerList.java:295)
          15:31:06 at com.microsoft.tfs.util.listeners.StandardListenerList$ListenerNode.addListener(StandardListenerList.java:304)
          15:31:06 at com.microsoft.tfs.util.listeners.StandardListenerList$ListenerNode.addListener(StandardListenerList.java:304)
          15:31:06 at com.microsoft.tfs.util.listeners.StandardListenerList$ListenerNode.addListener(StandardListenerList.java:304)
          ...

          Show
          peaz Mikael Norée added a comment - I have the same problem using Jenkins 2.20 and TFS-plugin 5.1.0. After a 2-3 days all jobs start failing with same message in output. Fixed by restarting Jenkins service on the windows server. 15:31:06 FATAL: null 15:31:06 java.lang.StackOverflowError 15:31:06 at com.microsoft.tfs.util.listeners.StandardListenerList$ListenerNode.addListener(StandardListenerList.java:295) 15:31:06 at com.microsoft.tfs.util.listeners.StandardListenerList$ListenerNode.addListener(StandardListenerList.java:304) 15:31:06 at com.microsoft.tfs.util.listeners.StandardListenerList$ListenerNode.addListener(StandardListenerList.java:304) 15:31:06 at com.microsoft.tfs.util.listeners.StandardListenerList$ListenerNode.addListener(StandardListenerList.java:304) ...
          Hide
          ssingh Supreet Singh added a comment -

          I am also facing similar issue with Jenkins ver. 1.642.18.3 (CloudBees Jenkins Enterprise 15.11) and TFS plugin 5.2.1. After every 3-4 days all jobs start failing on windows slave and need a restart of slave service to fix the issue.

          Does anyone has a permanent fix for this issue?

          Show
          ssingh Supreet Singh added a comment - I am also facing similar issue with Jenkins ver. 1.642.18.3 (CloudBees Jenkins Enterprise 15.11) and TFS plugin 5.2.1. After every 3-4 days all jobs start failing on windows slave and need a restart of slave service to fix the issue. Does anyone has a permanent fix for this issue?
          Hide
          adactuslatem Spenser Austin added a comment -

          This was supposed to be fixed in the Team Explorer Everywhere SDK v14.0.3 Release. This was included in version 5.1.0 of the TFS-plugin.

          I'm not sure why this is still happening, unless it wasn't actually fixed in the Team Explorer Anywhere 14.0.3.

          I'm on plugin version 5.3 and we still get this error occasionally.

          The only permanent fix would be to get it fixed in the TFS SDK.

          Show
          adactuslatem Spenser Austin added a comment - This was supposed to be fixed in the Team Explorer Everywhere SDK v14.0.3 Release . This was included in version 5.1.0 of the TFS-plugin . I'm not sure why this is still happening, unless it wasn't actually fixed in the Team Explorer Anywhere 14.0.3. I'm on plugin version 5.3 and we still get this error occasionally. The only permanent fix would be to get it fixed in the TFS SDK.
          Hide
          kelliejos Kellie Jos added a comment -

          Thank you for using the tfs-plugin. It appears that your issue was found in an older version of the plugin. There have been a number of improvements made in the recent releases that may have resolved your issue. Please update to the most recent 5.126.0 release. If your issue persists, please open a new ticket and we will investigate.

          Show
          kelliejos Kellie Jos added a comment - Thank you for using the tfs-plugin. It appears that your issue was found in an older version of the plugin. There have been a number of improvements made in the recent releases that may have resolved your issue. Please update to the most recent 5.126.0 release. If your issue persists, please open a new ticket and we will investigate.
          Hide
          flocki Florian Pepping added a comment -

          Hi all,

          unfortunately I can still confirm the bug/problem with the latest Jenkins and TFS-Versions.
          Environment is:

          • Windows Server 2012 R2
            Java HotSpot(TM) 64-Bit Server VM (1.8.0_151)
          • Jenkins ver. 2.121.1
          • TFS Plugin 5.133.0

          So this topic is not solved and still being pending. If some further input or traces are required please let me know. I'll try to provide them as far as possible.

          Thanks and best regards
          Flocki

          Show
          flocki Florian Pepping added a comment - Hi all, unfortunately I can still confirm the bug/problem with the latest Jenkins and TFS-Versions. Environment is: Windows Server 2012 R2 Java HotSpot(TM) 64-Bit Server VM (1.8.0_151) Jenkins ver. 2.121.1 TFS Plugin 5.133.0 So this topic is not solved and still being pending. If some further input or traces are required please let me know. I'll try to provide them as far as possible. Thanks and best regards Flocki
          Hide
          mmclaughlin Mitch McLaughlin added a comment -

          Kellie Jos

          I have just seen this recreated as well using:

          • Jenkins Version: 2.89.4.2
          • TFS Plugin Version: 5.133.0
          • Java Version: 1.8.0_181
          • Operating System: RedHat Linux 3.10.0-862.14.4.el7.x86_64

          Is there any plan to address this? Or a potential workaround to allow for jobs to continue running?

          Show
          mmclaughlin Mitch McLaughlin added a comment - Kellie Jos -  I have just seen this recreated as well using: Jenkins Version: 2.89.4.2 TFS Plugin Version: 5.133.0 Java Version: 1.8.0_181 Operating System: RedHat Linux 3.10.0-862.14.4.el7.x86_64 Is there any plan to address this? Or a potential workaround to allow for jobs to continue running?
          Hide
          jmiddour Joe Middour added a comment -

          Mitch,

          The workaround I settled on was to schedule a nightly job to restart Jenkins.

          This is the command I use:
          java -jar "%JENKINS_HOME%/war/WEB-INF/jenkins-cli.jar" -s "http://localhost" safe-restart --username <user> --password <pass>

          I recommend using one of the credential add-ins to protect the credentials for the job.

          Show
          jmiddour Joe Middour added a comment - Mitch, The workaround I settled on was to schedule a nightly job to restart Jenkins. This is the command I use: java -jar "%JENKINS_HOME%/war/WEB-INF/jenkins-cli.jar" -s "http://localhost" safe-restart --username <user> --password <pass> I recommend using one of the credential add-ins to protect the credentials for the job.

            People

            Assignee:
            jpricketmsft Jason Prickett
            Reporter:
            albertchu albert chu
            Votes:
            12 Vote for this issue
            Watchers:
            14 Start watching this issue

              Dates

              Created:
              Updated: