Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-35432

Scheduled builds crash after running for more than 8 hours

    XMLWordPrintable

Details

    Description

      I am using VS online, TFS plugin 4.1.

      I have scheduled builds (every 5 minutes) which checkcode code using the TFS plugin. After 1 or 2 days the builds start failing and I need to restart jenkins.

      From my initial searches, I believe this is an issue with how the plugin interacts with the TFS Java sdk.

      https://social.msdn.microsoft.com/Forums/vstudio/en-US/2ace3851-8e1c-4aee-8e2d-192b0810213a/tfs-sdk-leak?forum=tee

      The workaround is to close the TFSConfigurationServer first and then close the TFSTeamProjectCollection. Future versions of the TFS SDK will take care of closing the TFSConfigurationServer AND throw an exception if you close them the other way around.

      Here is the log:

      FATAL: null
      java.lang.StackOverflowError
      at com.microsoft.tfs.util.listeners.Comparators$IdentityComparator.compare(Comparators.java:57)
      at com.microsoft.tfs.util.listeners.StandardListenerList$ListenerNode.addListener(StandardListenerList.java:330)
      at com.microsoft.tfs.util.listeners.StandardListenerList$ListenerNode.addListener(StandardListenerList.java:343)
      at com.microsoft.tfs.util.listeners.StandardListenerList$ListenerNode.addListener(StandardListenerList.java:343)
      at com.microsoft.tfs.util.listeners.StandardListenerList$ListenerNode.addListener(StandardListenerList.java:343)
      at com.microsoft.tfs.util.listeners.StandardListenerList$ListenerNode.addListener(StandardListenerList.java:343)

      Attachments

        Activity

          peaz Mikael Norée added a comment -

          I have the same problem using Jenkins 2.20 and TFS-plugin 5.1.0. After a 2-3 days all jobs start failing with same message in output. Fixed by restarting Jenkins service on the windows server.

          15:31:06 FATAL: null
          15:31:06
          java.lang.StackOverflowError
          15:31:06 at com.microsoft.tfs.util.listeners.StandardListenerList$ListenerNode.addListener(StandardListenerList.java:295)
          15:31:06 at com.microsoft.tfs.util.listeners.StandardListenerList$ListenerNode.addListener(StandardListenerList.java:304)
          15:31:06 at com.microsoft.tfs.util.listeners.StandardListenerList$ListenerNode.addListener(StandardListenerList.java:304)
          15:31:06 at com.microsoft.tfs.util.listeners.StandardListenerList$ListenerNode.addListener(StandardListenerList.java:304)
          ...

          peaz Mikael Norée added a comment - I have the same problem using Jenkins 2.20 and TFS-plugin 5.1.0. After a 2-3 days all jobs start failing with same message in output. Fixed by restarting Jenkins service on the windows server. 15:31:06 FATAL: null 15:31:06 java.lang.StackOverflowError 15:31:06 at com.microsoft.tfs.util.listeners.StandardListenerList$ListenerNode.addListener(StandardListenerList.java:295) 15:31:06 at com.microsoft.tfs.util.listeners.StandardListenerList$ListenerNode.addListener(StandardListenerList.java:304) 15:31:06 at com.microsoft.tfs.util.listeners.StandardListenerList$ListenerNode.addListener(StandardListenerList.java:304) 15:31:06 at com.microsoft.tfs.util.listeners.StandardListenerList$ListenerNode.addListener(StandardListenerList.java:304) ...
          ssingh Supreet Singh added a comment -

          I am also facing similar issue with Jenkins ver. 1.642.18.3 (CloudBees Jenkins Enterprise 15.11) and TFS plugin 5.2.1. After every 3-4 days all jobs start failing on windows slave and need a restart of slave service to fix the issue.

          Does anyone has a permanent fix for this issue?

          ssingh Supreet Singh added a comment - I am also facing similar issue with Jenkins ver. 1.642.18.3 (CloudBees Jenkins Enterprise 15.11) and TFS plugin 5.2.1. After every 3-4 days all jobs start failing on windows slave and need a restart of slave service to fix the issue. Does anyone has a permanent fix for this issue?

          This was supposed to be fixed in the Team Explorer Everywhere SDK v14.0.3 Release. This was included in version 5.1.0 of the TFS-plugin.

          I'm not sure why this is still happening, unless it wasn't actually fixed in the Team Explorer Anywhere 14.0.3.

          I'm on plugin version 5.3 and we still get this error occasionally.

          The only permanent fix would be to get it fixed in the TFS SDK.

          adactuslatem Spenser Austin added a comment - This was supposed to be fixed in the Team Explorer Everywhere SDK v14.0.3 Release . This was included in version 5.1.0 of the TFS-plugin . I'm not sure why this is still happening, unless it wasn't actually fixed in the Team Explorer Anywhere 14.0.3. I'm on plugin version 5.3 and we still get this error occasionally. The only permanent fix would be to get it fixed in the TFS SDK.
          kelliejos Kellie Jos added a comment -

          Thank you for using the tfs-plugin. It appears that your issue was found in an older version of the plugin. There have been a number of improvements made in the recent releases that may have resolved your issue. Please update to the most recent 5.126.0 release. If your issue persists, please open a new ticket and we will investigate.

          kelliejos Kellie Jos added a comment - Thank you for using the tfs-plugin. It appears that your issue was found in an older version of the plugin. There have been a number of improvements made in the recent releases that may have resolved your issue. Please update to the most recent 5.126.0 release. If your issue persists, please open a new ticket and we will investigate.

          Hi all,

          unfortunately I can still confirm the bug/problem with the latest Jenkins and TFS-Versions.
          Environment is:

          • Windows Server 2012 R2
            Java HotSpot(TM) 64-Bit Server VM (1.8.0_151)
          • Jenkins ver. 2.121.1
          • TFS Plugin 5.133.0

          So this topic is not solved and still being pending. If some further input or traces are required please let me know. I'll try to provide them as far as possible.

          Thanks and best regards
          Flocki

          flocki Florian Pepping added a comment - Hi all, unfortunately I can still confirm the bug/problem with the latest Jenkins and TFS-Versions. Environment is: Windows Server 2012 R2 Java HotSpot(TM) 64-Bit Server VM (1.8.0_151) Jenkins ver. 2.121.1 TFS Plugin 5.133.0 So this topic is not solved and still being pending. If some further input or traces are required please let me know. I'll try to provide them as far as possible. Thanks and best regards Flocki

          kelliejos

          I have just seen this recreated as well using:

          • Jenkins Version: 2.89.4.2
          • TFS Plugin Version: 5.133.0
          • Java Version: 1.8.0_181
          • Operating System: RedHat Linux 3.10.0-862.14.4.el7.x86_64

          Is there any plan to address this? Or a potential workaround to allow for jobs to continue running?

          mmclaughlin Mitch McLaughlin added a comment - kelliejos -  I have just seen this recreated as well using: Jenkins Version: 2.89.4.2 TFS Plugin Version: 5.133.0 Java Version: 1.8.0_181 Operating System: RedHat Linux 3.10.0-862.14.4.el7.x86_64 Is there any plan to address this? Or a potential workaround to allow for jobs to continue running?
          jmiddour Joe Middour added a comment -

          Mitch,

          The workaround I settled on was to schedule a nightly job to restart Jenkins.

          This is the command I use:
          java -jar "%JENKINS_HOME%/war/WEB-INF/jenkins-cli.jar" -s "http://localhost" safe-restart --username <user> --password <pass>

          I recommend using one of the credential add-ins to protect the credentials for the job.

          jmiddour Joe Middour added a comment - Mitch, The workaround I settled on was to schedule a nightly job to restart Jenkins. This is the command I use: java -jar "%JENKINS_HOME%/war/WEB-INF/jenkins-cli.jar" -s "http://localhost" safe-restart --username <user> --password <pass> I recommend using one of the credential add-ins to protect the credentials for the job.

          People

            jpricketmsft Jason Prickett
            albertchu albert chu
            Votes:
            12 Vote for this issue
            Watchers:
            14 Start watching this issue

            Dates

              Created:
              Updated: