Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-15098

svnexternals causing issues with concurrent builds

    XMLWordPrintable

Details

    Description

      We're getting these quite often with concurrently running jobs that pull out of SVN:

      java.io.IOException: Unable to delete /var/lib/jenkins/jobs/[jobname]/svnexternals.txt
      at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:112)
      at hudson.XmlFile.write(XmlFile.java:174)
      at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:708)
      at hudson.model.AbstractProject.checkout(AbstractProject.java:1195)
      at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:576)
      at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:465)
      at hudson.model.Run.run(Run.java:1409)
      at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
      at hudson.model.ResourceController.execute(ResourceController.java:88)
      at hudson.model.Executor.run(Executor.java:238)

      Attachments

        Activity

          mlewicka Margaret Lewicka created issue -
          kutzi kutzi added a comment -

          Is this on Windows?

          kutzi kutzi added a comment - Is this on Windows?
          kutzi kutzi added a comment -

          No feedback from reporter, so closing as Incomplete

          kutzi kutzi added a comment - No feedback from reporter, so closing as Incomplete
          kutzi kutzi made changes -
          Field Original Value New Value
          Resolution Incomplete [ 4 ]
          Status Open [ 1 ] Resolved [ 5 ]
          saik Sai K added a comment -

          I am also seeing the same issue with concurrent builds of the same job on the same node.
          I am using Linux.

          saik Sai K added a comment - I am also seeing the same issue with concurrent builds of the same job on the same node. I am using Linux.
          saik Sai K made changes -
          Resolution Incomplete [ 4 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          iluxa86 Ilya Lebedev added a comment -

          I see the same issue using Linux. With 3 slaves running the same job simultenously.

          iluxa86 Ilya Lebedev added a comment - I see the same issue using Linux. With 3 slaves running the same job simultenously.
          aidan Aidan Lawn added a comment - - edited

          I see same issue with concurrent jobs.
          Jenkins version: Jenkins ver. 1.491
          Jenkins Subversion plugin version: 1.45 (was also occurring on 1.43 - I updated to latest but didn't help)

          As a workaround I am going to have to make a parent job that does the checkout first, then the concurrent jobs can run. Not ideal and may not be suitable for others situations.

          Note: We only see this issue under load, when I guess slower running checkouts makes the chance of collision higher.

          aidan Aidan Lawn added a comment - - edited I see same issue with concurrent jobs. Jenkins version: Jenkins ver. 1.491 Jenkins Subversion plugin version: 1.45 (was also occurring on 1.43 - I updated to latest but didn't help) As a workaround I am going to have to make a parent job that does the checkout first, then the concurrent jobs can run. Not ideal and may not be suitable for others situations. Note: We only see this issue under load, when I guess slower running checkouts makes the chance of collision higher.
          wael Wael Darwich added a comment -

          I am getting the same using Jenkins ver. 1.512 Linux master and Windows slaves

          wael Wael Darwich added a comment - I am getting the same using Jenkins ver. 1.512 Linux master and Windows slaves
          wael Wael Darwich added a comment -

          I found a solution that worked for me for this bug, in job configuration > Advanced Project Options > Retry Count > SCM checkout retry count for a value greater than zero, I tried 3 and it is working fine, I hope this helps others

          wael Wael Darwich added a comment - I found a solution that worked for me for this bug, in job configuration > Advanced Project Options > Retry Count > SCM checkout retry count for a value greater than zero, I tried 3 and it is working fine, I hope this helps others

          I have the same problem with jenkins LTS 1.509.1. on linux. The path "/var/lib/jenkins/jobs" shows that the svnexternals.txt is located on the master even if the builds are only executed on slaves. That means, every running build of this job uses the same svnexternals.txt at the same time. IMHO each build should have it's own svnexternals.txt or is there a reason, why there is only one for the whole thing?

          ffromm Frederik Fromm added a comment - I have the same problem with jenkins LTS 1.509.1. on linux. The path "/var/lib/jenkins/jobs" shows that the svnexternals.txt is located on the master even if the builds are only executed on slaves. That means, every running build of this job uses the same svnexternals.txt at the same time. IMHO each build should have it's own svnexternals.txt or is there a reason, why there is only one for the whole thing?
          oleg_nenashev Oleg Nenashev added a comment -

          The issue can be reproduced on the following environment:

          • subversion-1.50
          • jenkins 1.509.4 (master runs on RHEL6.4)
          • Windows slaves (seems it does not matter)

          Even if the workaround exists, the bug is quite painful.

          oleg_nenashev Oleg Nenashev added a comment - The issue can be reproduced on the following environment: subversion-1.50 jenkins 1.509.4 (master runs on RHEL6.4) Windows slaves (seems it does not matter) Even if the workaround exists, the bug is quite painful.
          oleg_nenashev Oleg Nenashev made changes -
          Assignee Kohsuke Kawaguchi [ kohsuke ] kutzi [ kutzi ]
          oleg_nenashev Oleg Nenashev made changes -
          Assignee kutzi [ kutzi ] Oleg Nenashev [ oleg_nenashev ]
          oleg_nenashev Oleg Nenashev added a comment - PR: https://github.com/jenkinsci/subversion-plugin/pull/55
          oleg_nenashev Oleg Nenashev added a comment -

          The PR is still waiting for Kutzi's review...

          oleg_nenashev Oleg Nenashev added a comment - The PR is still waiting for Kutzi's review...
          oleg_nenashev Oleg Nenashev made changes -
          Assignee Oleg Nenashev [ oleg_nenashev ] kutzi [ kutzi ]
          kutzi kutzi made changes -
          Due Date 2014-03-09
          kutzi kutzi made changes -
          Due Date 2014-03-09

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          src/main/java/hudson/scm/SubversionSCM.java
          src/main/java/hudson/scm/subversion/ExternalsFileManager.java
          http://jenkins-ci.org/commit/subversion-plugin/dc5e7a6c58c9d9d05cb89f1a56fa1f7eaf99a569
          Log:
          [FIXED JENKINS-15098] - Added synchronization of all reads/modifications of svnexternals.txt file.

          Resolves https://issues.jenkins-ci.org/browse/JENKINS-15098

          Signed-off-by: Oleg Nenashev <nenashev@synopsys.com>

          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: src/main/java/hudson/scm/SubversionSCM.java src/main/java/hudson/scm/subversion/ExternalsFileManager.java http://jenkins-ci.org/commit/subversion-plugin/dc5e7a6c58c9d9d05cb89f1a56fa1f7eaf99a569 Log: [FIXED JENKINS-15098] - Added synchronization of all reads/modifications of svnexternals.txt file. Resolves https://issues.jenkins-ci.org/browse/JENKINS-15098 Signed-off-by: Oleg Nenashev <nenashev@synopsys.com>
          scm_issue_link SCM/JIRA link daemon made changes -
          Resolution Fixed [ 1 ]
          Status Reopened [ 4 ] Resolved [ 5 ]

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          src/main/java/hudson/scm/SubversionSCM.java
          src/main/java/hudson/scm/SvnExternalsFileManager.java
          src/main/java/hudson/scm/subversion/ExternalsFileManager.java
          http://jenkins-ci.org/commit/subversion-plugin/9e45adf03dde5e24ec19f80f044c266656136875
          Log:
          JENKINS-15098 - Refactoring
          Moved base class to another package, modified visibility and javadoc.

          Related issues: https://issues.jenkins-ci.org/browse/JENKINS-15098

          Signed-off-by: Oleg Nenashev <nenashev@synopsys.com>

          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: src/main/java/hudson/scm/SubversionSCM.java src/main/java/hudson/scm/SvnExternalsFileManager.java src/main/java/hudson/scm/subversion/ExternalsFileManager.java http://jenkins-ci.org/commit/subversion-plugin/9e45adf03dde5e24ec19f80f044c266656136875 Log: JENKINS-15098 - Refactoring Moved base class to another package, modified visibility and javadoc. Related issues: https://issues.jenkins-ci.org/browse/JENKINS-15098 Signed-off-by: Oleg Nenashev <nenashev@synopsys.com>

          Code changed in jenkins
          User: Christoph Kutzinski
          Path:
          src/main/java/hudson/scm/SubversionSCM.java
          src/main/java/hudson/scm/SvnExternalsFileManager.java
          http://jenkins-ci.org/commit/subversion-plugin/23168bd2710b672ac81b3eb397402b8fc809d301
          Log:
          Merge pull request #65 from synopsys-arc-oss/externalsfile-conflicts

          [FIXED JENKINS-15098] - Access conflicts to svnexternals.txt

          Compare: https://github.com/jenkinsci/subversion-plugin/compare/cd7b24e71725...23168bd2710b

          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Christoph Kutzinski Path: src/main/java/hudson/scm/SubversionSCM.java src/main/java/hudson/scm/SvnExternalsFileManager.java http://jenkins-ci.org/commit/subversion-plugin/23168bd2710b672ac81b3eb397402b8fc809d301 Log: Merge pull request #65 from synopsys-arc-oss/externalsfile-conflicts [FIXED JENKINS-15098] - Access conflicts to svnexternals.txt Compare: https://github.com/jenkinsci/subversion-plugin/compare/cd7b24e71725...23168bd2710b
          ircbot Jenkins IRC Bot made changes -
          Component/s core [ 15593 ]
          Component/s concurrent-build [ 15628 ]
          rtyler R. Tyler Croy made changes -
          Workflow JNJira [ 145818 ] JNJira + In-Review [ 191641 ]

          People

            kutzi kutzi
            mlewicka Margaret Lewicka
            Votes:
            7 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: