• Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • core
    • Linux jenkins 2.6.32-43-server #97-Ubuntu SMP Wed Sep 5 16:56:41 UTC 2012 x86_64 GNU/Linux

      For more than a week, we have experienced deadlocks every morning.

      We suspect it occurs when someone hits the web ui for the first time that day. We do have nightly rebuilds, but the deadlock doesn't appear to occur until much later in the morning. The deadlocks don't occur at the same time, but they are consistent every morning.

      We have tried a couple things so far:
      Upgrade from 1.520 to 1.525
      Replace Winstone with Jetty (suggested by nanoyme #jenkins@freenode)

      I've attached the jstack traces for when jenkins gets wedged and after a fresh restart as well as the plugins list.

      Would like some pointers on what to diagnose next. We were hoping replacing winstone with jetty was going to work.

          [JENKINS-19215] Consistent daily morning deadlocks

          milki mlk added a comment -

          With jetty, its wedged twice today already.

          milki mlk added a comment - With jetty, its wedged twice today already.

          milki mlk added a comment -

          I spun up another instance of jenkins using the same setup (new isntances of jetty + a copy of JENKINS_HOME) except I disabled two plugins:

          Email-ext plugin 2.30.2
          Jenkins IRC Plugin 2.23

          I discovered in the All builds graph that the original instance failed to do most of the nightly builds while the test instance did. Again the original instance wedged in the morning. Under jetty, we can see the number of processes (via ganglia) rise consistently after the first batch of nightlies. We haven't discovered any abnormal behaviour under winstone.

          milki mlk added a comment - I spun up another instance of jenkins using the same setup (new isntances of jetty + a copy of JENKINS_HOME) except I disabled two plugins: Email-ext plugin 2.30.2 Jenkins IRC Plugin 2.23 I discovered in the All builds graph that the original instance failed to do most of the nightly builds while the test instance did. Again the original instance wedged in the morning. Under jetty, we can see the number of processes (via ganglia) rise consistently after the first batch of nightlies. We haven't discovered any abnormal behaviour under winstone.

          milki mlk added a comment -

          Falling back to email-ext 2.27, the version before everything started wedging, did not prevent the deadlock behaviour.

          milki mlk added a comment - Falling back to email-ext 2.27, the version before everything started wedging, did not prevent the deadlock behaviour.

          Following an update from version 1.411 to version 1.530, I am also experiencing that issue.

          In the morning, the web interface does not respond, it loads indefinitely. However, I have not experienced any missed job.

          Julien Froment Cousineau added a comment - Following an update from version 1.411 to version 1.530, I am also experiencing that issue. In the morning, the web interface does not respond, it loads indefinitely. However, I have not experienced any missed job.

          milki mlk added a comment -

          Upgrading to Java 7u15 has alleviated the daily lockups. It has now become a weekly lockup.

          milki mlk added a comment - Upgrading to Java 7u15 has alleviated the daily lockups. It has now become a weekly lockup.

          milki mlk added a comment - - edited

          Possibly related, looks like the irc plugin actually killed the winstone server. See attached irbot.kills.winstone log

          milki mlk added a comment - - edited Possibly related, looks like the irc plugin actually killed the winstone server. See attached irbot.kills.winstone log

          milki mlk added a comment -

          With the Java Melody monitoring plugin, we have concluded that memory is not an issue immediately before Jenkins freezes.

          milki mlk added a comment - With the Java Melody monitoring plugin, we have concluded that memory is not an issue immediately before Jenkins freezes.

          Oleg Nenashev added a comment -

          In https://issues.jenkins-ci.org/browse/JENKINS-20051 winstone also goes down, but the issue has been caused by Outage of File Descriptors and LDAP issues w/o any deadlocks.

          Seems that automatic failover of a web container could be useful.

          Oleg Nenashev added a comment - In https://issues.jenkins-ci.org/browse/JENKINS-20051 winstone also goes down, but the issue has been caused by Outage of File Descriptors and LDAP issues w/o any deadlocks. Seems that automatic failover of a web container could be useful.

          Daniel Beck added a comment -

          Is this still a problem for you?

          Daniel Beck added a comment - Is this still a problem for you?

          milki mlk added a comment -

          Upgrading Java to 7u55, increasing PermGen to 128m, and adding a ton more ram seems to do the trick.

          The irc plugin can still kill winstone and there are still File Descriptor leaks. But Jenkins no longer consistently deadlocks at least.

          milki mlk added a comment - Upgrading Java to 7u55, increasing PermGen to 128m, and adding a ton more ram seems to do the trick. The irc plugin can still kill winstone and there are still File Descriptor leaks. But Jenkins no longer consistently deadlocks at least.

            Unassigned Unassigned
            milki milki mlk
            Votes:
            2 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: