Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-60898

Contention on fingerprint files when using Docker steps

    XMLWordPrintable

Details

    Description

      We have a jenkins master and slave running several docker based jobs every minute.

      The execution times of each job is well under a minute, the longest one is 45 seconds.

      Each job gets to https://github.com/jenkinsci/docker-workflow-plugin/blob/0540c1482c4b18beaa0f036491d5a732f0e024b0/src/main/java/org/jenkinsci/plugins/docker/workflow/WithContainerStep.java#L196 and hangs.

      Looking at the thread dump (attached) threads are blocking on hudson.model.Fingerprint.save(Fingerprint.java:1244).

      Further investigation yielded that these saves are writing to the jenkins fingerprint files.

      Looking at the master server, the fingerprints directory is varying by hundreds of MBs every second.

      There are several large xml files being created in the subdirectories containing ContainerRecords many for containers which are not currently returned by docker ps -a.

      The number of build logs to keep is set to 25.

      The disk used by the jenkins master is getting hammered more than several high usage production databases we have running in production (image attached).

       

      Attachments

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-60570 Docker fingerprint data written to file from WithContainerStep.java never gets cleaned up when image is used repeatedly

          • Major - Major loss of function.
          • Open
          relates to

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-49235 Fingerprinting added in ssh-slaves causes memory-leak and performance issue with dynamic slaves

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Activity

            Ascending order - Click to sort in descending order
            jlundy Jeremy Lundy added a comment -

            Similar issue was found in the credentials plugin

            jlundy Jeremy Lundy added a comment - Similar issue was found in the credentials plugin
            raihaan Raihaan Shouhell added a comment -

            Latest version of the docker-workflow plugin seems to have removed that fingerprinting

            raihaan Raihaan Shouhell added a comment - Latest version of the docker-workflow plugin seems to have removed that fingerprinting
            qmnonic Matt MacGillivray added a comment -

            raihaan There is no `docker-workflow` plugin?  Not sure what to do to get this fix?

            qmnonic Matt MacGillivray added a comment - raihaan There is no `docker-workflow` plugin?  Not sure what to do to get this fix?
            raihaan Raihaan Shouhell added a comment -

            Thats docker pipeline plugin its shortname is docker-workflow-plugin

            raihaan Raihaan Shouhell added a comment - Thats docker pipeline plugin its shortname is docker-workflow-plugin

            People

              Unassigned Unassigned
              jlundy Jeremy Lundy
              Votes:
              2 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated: