Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-53140

Intermittent Random Stage Failures - No Changes Detected - Parallel Stages

    XMLWordPrintable

Details

    Description

      On a new Jenkins server, we're setting up new Jenkinsfiles from scratch.  We're using parallel stages, which seems like it might be a factor based on searches for related issues. 

      We have about 15 of these jobs with 4 parallel stages.  For about 1 in 10 jobs, a stage will randomly fail in 1 second or less.  If you hover over the stage it shows: 

      Failed to parse /var/lib/jenkins/jobs/MyOrg/jobs/Projects/jobs/myProject/builds/252/changelog3.xml

      If you click the Stage Log it says: 

      No changes for <my_scm_url> since the previous build`

      This can happen when the build is triggered by either of the following means:

      • From an upstream job
      • Manually by clicking "build now"

       

      Of note, we use SVN as our SCM with the following configuration:

      We use "Jenkinsfile from SCM", using URL like

      http://myserver/myrepo/_automation

      In the GUI, we do not configure SCM polling 

      In the Jenkinsfile, we configure SCM polling, every 2 minutes

      In the Jenkinsfile, we configure SCM checkout of 

      http://myserver/myrepo`|http://myserver/myrepo

       

      The reason I include our SVN/SCM backstory is because we're also seeing another anomaly related to SCM which might be relevant. . 

      For each SCM polling period, the Subversion Polling Log show approximately multiple entries all at the same timestamp.   These entries are some randomized mix of polling the two relevant URL's :

      http://myserver/myrepo

      and 

      http://myserver/myrepo/_automation

      Interestingly, we've figured something out about these duplicate "Subversion Polling Entries" . There is 1 entry checking

      http://myserver/myrepo/_automation

      for changes for each stage in the build for the ". So, if we have 1 "Checkout Stage", 1 "Parallel Stages" block, and 3 "Parallel Stages", we see 5 total "Subversion Polling Entries" for

      http://myserver/myrepo/_automation

      . Not sure if this is by design or not.

       

       

       

       

       

      We have the latest versions of all plugins. 

      Attachments

        Issue Links

          Activity

            solvingj jerry wiltse added a comment -

            Is there any other report of a similar issue? Are we the only ones having this problem with parallel jobs? Maybe few people still use SVN Plugin? 

            solvingj jerry wiltse added a comment - Is there any other report of a similar issue? Are we the only ones having this problem with parallel jobs? Maybe few people still use SVN Plugin? 
            solvingj jerry wiltse added a comment -

            Also of note, I've discovered that we'll never really be able to use: 

            skip_default_checkout()

            as any part of a workaround because then we don't get

            SVN_REVISION_1
            and
            SVN_REPOSITORY_URL

            environment variable which we require.

            solvingj jerry wiltse added a comment - Also of note, I've discovered that we'll never really be able to use:  skip_default_checkout() as any part of a workaround because then we don't get SVN_REVISION_1 and SVN_REPOSITORY_URL environment variable which we require.
            solvingj jerry wiltse added a comment -

            I have some new information that might be relevant to isolating the problem. I moved the pipline out of jenkinsfile and into the GUI. The duplicates in the changelog have gone away. 

            Thus, the recipe for at least one problem is simple: 

            • Pipeline from SCM
            • SVN being the SCM used for this
            • Multiple Stages?

            Unfortunately, we can't currently find a workaround here without more help.  It seems that when not using "pipeline script from SCM", the stages all run in different subdirectories rather than the directory the "checkout" stage ran in.  For some reason, the first stage after "checkout" runs in the same directory, but the others choose different directories.  our existing scripts all count on the files being there from a checkout. 

             

            Also, obviously, we want to use "pipeline script from scm", and it's really crazy if we cant. 

            Thanks again for any feedback. 

             

            solvingj jerry wiltse added a comment - I have some new information that might be relevant to isolating the problem. I moved the pipline out of jenkinsfile and into the GUI. The duplicates in the changelog have gone away.  Thus, the recipe for at least one problem is simple:  Pipeline from SCM SVN being the SCM used for this Multiple Stages? Unfortunately, we can't currently find a workaround here without more help.  It seems that when not using "pipeline script from SCM", the stages all run in different subdirectories rather than the directory the "checkout" stage ran in.  For some reason, the first stage after "checkout" runs in the same directory, but the others choose different directories.  our existing scripts all count on the files being there from a checkout.    Also, obviously, we want to use "pipeline script from scm", and it's really crazy if we cant.  Thanks again for any feedback.   
            abayer Andrew Bayer added a comment -

            Not sure if this is a complete duplicate of JENKINS-34313, but it's definitely similar.

            abayer Andrew Bayer added a comment - Not sure if this is a complete duplicate of JENKINS-34313 , but it's definitely similar.
            solvingj jerry wiltse added a comment -

            Yes, we can close this because this issue was resolved in that ticket.

            solvingj jerry wiltse added a comment - Yes, we can close this because this issue was resolved in that ticket.

            People

              Unassigned Unassigned
              solvingj jerry wiltse
              Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: