Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-50919

Ephemeral/Cloud Slaves History Present (in massive quantities)

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Minor
    • Resolution: Unresolved
    • Labels:
    • Environment:
      CloudBees Jenkins Enterprise 2.89.4.2-rolling; changelog-history-plugin 2.18
    • Similar Issues:

      Description

      While we weren't paying close attention, the changelog-history-plugin created 1 million entries in the config-history/nodes folder. The reason is apparently that we have been allocating slaves from a cloud/swarm that live for a few minutes and then are reclaimed.

      For example here is a snippet of some of the directories created:

      ./docker-8cd90563e4bb5_deleted_20180420_201419_896
      ./docker-8ccdf12fbb215_deleted_20180420_200757_598
      ./docker-8c9b06442f1b3_deleted_20180420_190621_902
      ./docker-8cdbb299a77ee
      ./docker-8cdd7cf907080
      ./docker-8cb4634d8051c_deleted_20180420_193442_663
      ./docker-8c97a99997968_deleted_20180420_190313_641
      ./docker-8cb8ebbab1aa8_deleted_20180420_193606_401
      ./docker-8cbdc8cfa9186_deleted_20180420_195041_264
      ./docker-8c8bfc4dfd5ba_deleted_20180420_184741_165
      ./docker-8cd3b4d6da0ed_deleted_20180420_200527_843
      ./docker-8c8dee6436c49_deleted_20180420_184730_195
      ./docker-8cafb29f67d1c_deleted_20180420_192601_188
      ./docker-8cd90560b2493_deleted_20180420_201451_943
      ./docker-8cbddff5b52a6_deleted_20180420_194238_688
      ./docker-8caca89cd5a6f_deleted_20180420_192758_912
      ./docker-8cbaf4fff12fd_deleted_20180420_194208_624
      ./docker-8c8ba6d61cff9_deleted_20180420_184756_715

      The release notes for this plugin contain the following comments:
      {{}}

      Version 2.11 (April 17 2015)

      • Avoid calling User.current() during Jenkins initialization (Thanks to Thomas de Grenier de Latour)
      • Avoid tracking changes for cloud slave as well (Thanks to Ryan Campbel)

      Version 2.10 (November 12 2014)

      • Fix proposal for NPE in ComputerHistoryListener.onConfigurationChange method (Thanks to William Bernardet)
      • (JENKINS-22639) Don't record changes to AbstractCloudSlaves or Ephemeral Nodes (Thanks to Ryan Campbell and Jesse Click)

      I suspect either these changes were reverted somehow, or it is not covering our case.

      We are worried that this behavior may be one of the reasons why:

      1. Our lockable resources / throttling is no longer working and 
      2. we end up with apparent scheduler deadlocks (spun up slaves cannot be connected to before they get reclaimed).

      These last two items are pure conjecture, but based on these JIRAs:

      https://issues.jenkins-ci.org/browse/JENKINS-22639
      https://issues.jenkins-ci.org/browse/JENKINS-41844

        Attachments

          Activity

          Hide
          stevenatcisco Steven Christenson added a comment -

          On our primary server, 3000 config-history/nodes/deleted/ folders were created in 17 hours - about 3 a minute.

          Show
          stevenatcisco Steven Christenson added a comment - On our primary server, 3000 config-history/nodes/ deleted / folders were created in 17 hours - about 3 a minute.

            People

            Assignee:
            escoem Emilio Escobar
            Reporter:
            stevenatcisco Steven Christenson
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated: