Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-60898

Contention on fingerprint files when using Docker steps

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      We have a jenkins master and slave running several docker based jobs every minute.

      The execution times of each job is well under a minute, the longest one is 45 seconds.

      Each job gets to https://github.com/jenkinsci/docker-workflow-plugin/blob/0540c1482c4b18beaa0f036491d5a732f0e024b0/src/main/java/org/jenkinsci/plugins/docker/workflow/WithContainerStep.java#L196 and hangs.

      Looking at the thread dump (attached) threads are blocking on hudson.model.Fingerprint.save(Fingerprint.java:1244).

      Further investigation yielded that these saves are writing to the jenkins fingerprint files.

      Looking at the master server, the fingerprints directory is varying by hundreds of MBs every second.

      There are several large xml files being created in the subdirectories containing ContainerRecords many for containers which are not currently returned by docker ps -a.

      The number of build logs to keep is set to 25.

      The disk used by the jenkins master is getting hammered more than several high usage production databases we have running in production (image attached).

       

        Attachments

          Issue Links

            Activity

            jlundy Jeremy Lundy created issue -
            jlundy Jeremy Lundy made changes -
            Field Original Value New Value
            Link This issue relates to JENKINS-49235 [ JENKINS-49235 ]
            jlundy Jeremy Lundy made changes -
            Attachment JenkinsThreadDump.txt [ 50055 ]
            jlundy Jeremy Lundy made changes -
            Attachment JenkinsThreadDump.txt [ 50056 ]
            qmnonic Matt MacGillivray made changes -
            Link This issue is related to JENKINS-60570 [ JENKINS-60570 ]

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              jlundy Jeremy Lundy
              Votes:
              2 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated: