Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-71768

Race condition when copying archived artifacts in another job

XMLWordPrintable

      In a Multijob setup, we create and archive some artifacts in a sub job, which we afterwards copy into the parent job to be consumed during a "docker build" step (which directly follows the "copy artifacts" step).

      Before switching to S3 artifact manager, everything went fine, but afterwards the jobs started to fail because the "docker build" step couldn't find the artifacts Jenkins said it had copied successfully.

      We could reproduce this in a test job. There is some delay between "artifacts copied" and "artifacts being accessible", and the artifact in our case was just 15kB.

      This renders the plugin unusable (for us), hence the "Blocker".

          [JENKINS-71768] Race condition when copying archived artifacts in another job

          Dirk Heinrichs created issue -

          Dirk Heinrichs added a comment - - edited
          1. This nice but I guess the delay would vary depending on artifact size, right?
          2. This shouldn't be needed at all, since that's what I would expect from the plugin in the first place. If it says they have been copied, they should have been copied.
          3. Multijobs don't have stash/unstash, that's a pipeline thing.

          Dirk Heinrichs added a comment - - edited This nice but I guess the delay would vary depending on artifact size, right? This shouldn't be needed at all, since that's what I would expect from the plugin in the first place. If it says they have been copied, they should have been copied. Multijobs don't have stash/unstash, that's a pipeline thing.
          Mark Waite made changes -
          Comment [ Hello,

          Understanding the Issue

          You're encountering a delay in artifact accessibility after copying them from a sub-job to the parent job in a Multijob setup using the S3 artifact manager. This delay is causing the subsequent "docker build" step to fail.

          Potential Solutions [aarpmahjongg|https://www.aarp-mahjongg.com/]
           # Increase Artifact Copy Delay:
          Introduce a deliberate delay (e.g., using a sleep command or a timer) after the "copy artifacts" step to allow sufficient time for the artifact to become accessible.
          Monitor the artifact's availability using scripting or API calls before proceeding with the "docker build" step.
           # Leverage Artifact Manager APIs:
          Explore the S3 artifact manager's APIs to check the artifact's status or metadata to determine if it's ready for consumption.
          Use this information to conditionally proceed with the "docker build" step.
           # Utilize Stash/Unstash:
          Instead of using the "copy artifacts" step, consider using the stash and unstash steps. This might provide more reliable artifact management within the same job.
          Stash the artifacts in the sub-job and unstash them in the parent job before the "docker build" step. ]
          Mark Waite made changes -
          Comment [ Spam comment ]
          Mark Waite made changes -
          Comment [ Spam comment ]
          Mark Waite made changes -
          Comment [ spam comment ]

            Unassigned Unassigned
            dhs Dirk Heinrichs
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated: