Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-58068

Races / corruption on revision.txt in pipeline-parallel

    XMLWordPrintable

Details

    Description

      When used in a pipleine parallel block, the subversion plugin suffers from raceconditions due to the dependency on the "revision.txt" file as part of the checkout and change log computation process.

      Unsynchronized access to "revision.txt" causes `SubversionChangeLogBuilder ` to fail spuriously, as the file may be written to concurrently by other running checkout processes.

      The final "revision.txt" eventually reaches a stable state where it only contains the revision of the last "checkout" step, but in the process `SubversionChangeLogBuilder` has failed to record changes repeatedly (bailout with "No revision found for ....") and the "build.xml" is corrupted and contains fragments like the following:

          <org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState plugin="workflow-scm-step@2.9">
            <revisionStates>
              <entry>
                <string>svn http://redacted/trunk</string>
                <hudson.scm.SVNRevisionState plugin="subversion@2.12.1">
                  <revisions>
                    <entry>
                      <string>ed/trunk</string>
                      <long>16122</long>
                    </entry>
                    <entry>
                      <string>svn http://redacted/trunk</string>
                      <long>12362</long>
                    </entry>
                  </revisions>
                </hudson.scm.SVNRevisionState>
              </entry>
            </revisionStates>
          </org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState>
      

      In summary:

      • Unsafe reads from "revision.txt" which contains data from parallel checkout processes
      • Lost writes to "revision.txt" from previous checkout steps in the same build, with unstable results

      While producing "revision.txt" as a side product of the build process may be desired to keep as a side effect, using it internally means state between `SubversionSCM` instances in the same build is shared in an unsafe way. Assuming it is only written to once per build may be somewhat fitting to Freestyle projects, but is unsuited for pipeline projects.

      Attachments

        Activity

          ext3h Andreas Ringlstetter created issue -
          ext3h Andreas Ringlstetter made changes -
          Field Original Value New Value
          Link This issue is related to JENKINS-39968 [ JENKINS-39968 ]
          ext3h Andreas Ringlstetter made changes -
          Description In the scope of JENKINS-39968, the subversion plugin suffers from additional races due to the dependency on the "revision.txt" file as part of the checkout and change log computation process.

          Unsynchronized access to "revision.txt" causes `SubversionChangeLogBuilder ` to fail spuriously, as the file may be written to concurrently by other running checkout processes.

          The final "revision.txt" eventually reaches a stable state where it only contains the revision of the last "checkout" step, but in the process `SubversionChangeLogBuilder` has failed to record changes repeatedly (bailout with "No revision found for ....") and the "build.xml" is corrupted and contains fragments like the following:

          {code:java}
              <org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState plugin="workflow-scm-step@2.9">
                <revisionStates>
                  <entry>
                    <string>svn http://redacted/trunk&lt;/string>
                    <hudson.scm.SVNRevisionState plugin="subversion@2.12.1">
                      <revisions>
                        <entry>
                          <string>ed/trunk</string>
                          <long>16122</long>
                        </entry>
                        <entry>
                          <string>svn http://redacted/trunk&lt;/string>
                          <long>12362</long>
                        </entry>
                      </revisions>
                    </hudson.scm.SVNRevisionState>
                  </entry>
                </revisionStates>
              </org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState>
          {code}

          In summary:
          * Unsafe reads from "revision.txt" which contains data from parallel checkout processes
          * Lost writes to "revision.txt" from previous checkout steps in the same build, with unstable results

          While producing "revision.txt" as a side product of the build process may be desired to keep as a side effect, using it internally means state between `SubversionSCM` instances in the same build is shared in an unsafe way.
          In the scope of JENKINS-39968, the subversion plugin suffers from additional races due to the dependency on the "revision.txt" file as part of the checkout and change log computation process.

          Unsynchronized access to "revision.txt" causes `SubversionChangeLogBuilder ` to fail spuriously, as the file may be written to concurrently by other running checkout processes.

          The final "revision.txt" eventually reaches a stable state where it only contains the revision of the last "checkout" step, but in the process `SubversionChangeLogBuilder` has failed to record changes repeatedly (bailout with "No revision found for ....") and the "build.xml" is corrupted and contains fragments like the following:

          {code:java}
              <org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState plugin="workflow-scm-step@2.9">
                <revisionStates>
                  <entry>
                    <string>svn http://redacted/trunk&lt;/string>
                    <hudson.scm.SVNRevisionState plugin="subversion@2.12.1">
                      <revisions>
                        <entry>
                          <string>ed/trunk</string>
                          <long>16122</long>
                        </entry>
                        <entry>
                          <string>svn http://redacted/trunk&lt;/string>
                          <long>12362</long>
                        </entry>
                      </revisions>
                    </hudson.scm.SVNRevisionState>
                  </entry>
                </revisionStates>
              </org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState>
          {code}

          In summary:
          * Unsafe reads from "revision.txt" which contains data from parallel checkout processes
          * Lost writes to "revision.txt" from previous checkout steps in the same build, with unstable results

          While producing "revision.txt" as a side product of the build process may be desired to keep as a side effect, using it internally means state between `SubversionSCM` instances in the same build is shared in an unsafe way. Assuming it is only written to once per build may be somewhat fitting to Freestyle projects, but is unsuited for pipeline projects.
          ext3h Andreas Ringlstetter made changes -
          Link This issue is related to JENKINS-39968 [ JENKINS-39968 ]
          ext3h Andreas Ringlstetter made changes -
          Description In the scope of JENKINS-39968, the subversion plugin suffers from additional races due to the dependency on the "revision.txt" file as part of the checkout and change log computation process.

          Unsynchronized access to "revision.txt" causes `SubversionChangeLogBuilder ` to fail spuriously, as the file may be written to concurrently by other running checkout processes.

          The final "revision.txt" eventually reaches a stable state where it only contains the revision of the last "checkout" step, but in the process `SubversionChangeLogBuilder` has failed to record changes repeatedly (bailout with "No revision found for ....") and the "build.xml" is corrupted and contains fragments like the following:

          {code:java}
              <org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState plugin="workflow-scm-step@2.9">
                <revisionStates>
                  <entry>
                    <string>svn http://redacted/trunk&lt;/string>
                    <hudson.scm.SVNRevisionState plugin="subversion@2.12.1">
                      <revisions>
                        <entry>
                          <string>ed/trunk</string>
                          <long>16122</long>
                        </entry>
                        <entry>
                          <string>svn http://redacted/trunk&lt;/string>
                          <long>12362</long>
                        </entry>
                      </revisions>
                    </hudson.scm.SVNRevisionState>
                  </entry>
                </revisionStates>
              </org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState>
          {code}

          In summary:
          * Unsafe reads from "revision.txt" which contains data from parallel checkout processes
          * Lost writes to "revision.txt" from previous checkout steps in the same build, with unstable results

          While producing "revision.txt" as a side product of the build process may be desired to keep as a side effect, using it internally means state between `SubversionSCM` instances in the same build is shared in an unsafe way. Assuming it is only written to once per build may be somewhat fitting to Freestyle projects, but is unsuited for pipeline projects.
          When used in a pipleine parallel block, the subversion plugin suffers from raceconditions due to the dependency on the "revision.txt" file as part of the checkout and change log computation process.

          Unsynchronized access to "revision.txt" causes `SubversionChangeLogBuilder ` to fail spuriously, as the file may be written to concurrently by other running checkout processes.

          The final "revision.txt" eventually reaches a stable state where it only contains the revision of the last "checkout" step, but in the process `SubversionChangeLogBuilder` has failed to record changes repeatedly (bailout with "No revision found for ....") and the "build.xml" is corrupted and contains fragments like the following:

          {code:java}
              <org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState plugin="workflow-scm-step@2.9">
                <revisionStates>
                  <entry>
                    <string>svn http://redacted/trunk&lt;/string>
                    <hudson.scm.SVNRevisionState plugin="subversion@2.12.1">
                      <revisions>
                        <entry>
                          <string>ed/trunk</string>
                          <long>16122</long>
                        </entry>
                        <entry>
                          <string>svn http://redacted/trunk&lt;/string>
                          <long>12362</long>
                        </entry>
                      </revisions>
                    </hudson.scm.SVNRevisionState>
                  </entry>
                </revisionStates>
              </org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState>
          {code}

          In summary:
          * Unsafe reads from "revision.txt" which contains data from parallel checkout processes
          * Lost writes to "revision.txt" from previous checkout steps in the same build, with unstable results

          While producing "revision.txt" as a side product of the build process may be desired to keep as a side effect, using it internally means state between `SubversionSCM` instances in the same build is shared in an unsafe way. Assuming it is only written to once per build may be somewhat fitting to Freestyle projects, but is unsuited for pipeline projects.
          ifernandezcalvo Ivan Fernandez Calvo made changes -
          Assignee Ivan Fernandez Calvo [ ifernandezcalvo ]
          dd Dieter Dobbelaere added a comment - - edited

          We have experienced the exact same issue in our parallel pipelines (we could also track it down to corruption of "revision.txt"). This issue has plagued us with lots of spurious SCM triggered builds, which were not caused by real SCM changes, but by corrupted internal state files.

          Our workaround was to define a new "trigger" pipeline whose sole purpose it to check out all the sources (subversion + git repos) in a single stage. This "trigger" pipeline is triggered by SCM changes and triggers the main pipeline after completion.

          This solves the issue of spurious SCM triggered builds ("revision.txt" and "build.xml" are still corrupted in the main pipeline folder though).

          If possible, an update of the subversion-plugin would be most welcome though, such that this artificial construction can be avoided.

          dd Dieter Dobbelaere added a comment - - edited We have experienced the exact same issue in our parallel pipelines (we could also track it down to corruption of "revision.txt"). This issue has plagued us with lots of spurious SCM triggered builds, which were not caused by real SCM changes, but by corrupted internal state files. Our workaround was to define a new "trigger" pipeline whose sole purpose it to check out all the sources (subversion + git repos) in a single stage. This "trigger" pipeline is triggered by SCM changes and triggers the main pipeline after completion. This solves the issue of spurious SCM triggered builds ("revision.txt" and "build.xml" are still corrupted in the main pipeline folder though). If possible, an update of the subversion-plugin would be most welcome though, such that this artificial construction can be avoided.

          People

            Unassigned Unassigned
            ext3h Andreas Ringlstetter
            Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: