Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-54187

EC2 Plugin deadlock leaving Jenkins unresponsive

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • ec2-plugin
    • None

      We recently upgraded our Jenkins instance to the latest LTS (2.138.2) and started noticing some irregularities with the EC2 plugin, namely:

      • A lot more build nodes were created than before, for the same amount of incoming items in the build queue
      • This caused an increased amount of AWS API calls, namely StopInstances event. Possibly related to the previous point.

      This led us to try out the latest snapshot, 1.41-SNAPSHOT build on revision d4bdd6b83a7102330fd97ffbbd067edc34e47f97. A few hours later, our Jenkins instance had a deadlock problem that is described in the log I'm attaching.

      Notice how the Gerrit plugin stops processing events after the EC2 plugin deadlock, this ultimately left our Jenkins unresponsive.

      One important bit of information might be that we're using the Stop/Disconnect on Idle Timeout plugin option.

      We're happy to provide more information if needed.

        1. all-jenkins.log
          26 kB
          Renzo Crisóstomo
        2. jenkins.log
          10 kB
          Danny
        3. jenkins.txt
          6 kB
          Callum Pember

            thoulen FABRIZIO MANFREDI
            ruenzuo Renzo Crisóstomo
            Votes:
            6 Vote for this issue
            Watchers:
            19 Start watching this issue

              Created:
              Updated:
              Resolved: