Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-27263

Checkout of svn:externals fails silently and intermittently

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • subversion-plugin
    • None
    • Jenkins ver. 1.583
      Subversion Plugin 2.4.3
      Sun Java 1.6.0_20 (on the client)

      We have configurations based on svn:externals for our product. Sometimes (apparently at random) checkouts fail silently leading to files missing in the build tree.

      Example of output during checkout (somewhat sanitized):

      00:00:00.002 - Started by an SCM change
      00:00:00.003 - Building remotely on Jenkins Buildhost (i-461077a2) (linux) in workspace /mnt/workspace/8.30_Standard_Linux_build
      00:00:00.438 - Checking out a fresh workspace because /mnt/workspace/8.30_Standard_Linux_build/build doesn't exist
      00:00:00.476 - Cleaning local Directory build
      00:00:00.477 - Checking out <svn-url-for-config> at revision '2015-03-06T08:53:12.714 +0000'
      00:00:02.435 - AU Global_config_file
      00:00:02.435 - U .
      00:00:02.435 - Fetching '<svn-url-for-module-1>' at -1 into '/mnt/workspace/8.30_Standard_Linux_build/build/module1'

      ... (several modules working well)

      00:00:15.182 - At revision 48819
      00:00:15.182 - Fetching '<svn-url-for-module-5>' at -1 into '/mnt/workspace/8.30_Standard_Linux_build/build/module4'
      00:00:16.182 - A module5/dir_1

      ... (many files and dirs working)

      00:01:31.865 - A module5/dir_4
      00:01:31.865 - AU module5/dir_4/file_1.xml

      ... (working files)

      00:01:48.522 - A module5/dir_4/file_100.xml
      00:01:51.525 - Fetching '<svn-url-for-module-6>' at -1 into '/mnt/workspace/8.30_Standard_Linux_build/build/module5'
      00:01:53.522 - A module6/dir1

      ... (the rest of the checkout working)

      On the next build, we get this:

      00:00:00.021 - Started by an SCM change
      00:00:00.023 - Building remotely on Jenkins Buildhost (i-461077a2) (linux) in workspace /mnt/workspace/8.30_Standard_Linux_build
      00:00:00.489 - Updating <svn-url-for-config> at revision '2015-03-06T09:18:17.980 +0000'
      00:00:01.529 - Fetching '<svn-url-for-module-1>' at -1 into '/mnt/workspace/8.30_Standard_Linux_build/build/module1'
      00:00:01.784 - At revision 48825
      00:00:02.072 - Fetching '<svn-url-for-module-2>' at -1 into '/mnt/workspace/8.30_Standard_Linux_build/build/module2'
      00:00:02.324 - At revision 48825
      00:00:02.615 - Fetching '<svn-url-for-module-3>' at -1 into '/mnt/workspace/8.30_Standard_Linux_build/build/module3'
      00:00:03.232 - At revision 48825
      00:00:03.559 - Fetching '<svn-url-for-module-4>' at -1 into '/mnt/workspace/8.30_Standard_Linux_build/build/module4'
      00:00:04.162 - At revision 48825
      00:00:04.496 - Fetching '<svn-url-for-module-5>' at -1 into '/mnt/workspace/8.30_Standard_Linux_build/build/module5'
      00:00:07.958 - AU module5/dir_4/file_1.xml
      00:00:08.046 - A module5/dir_4/file_2.xml

      ... (The rest of module5 checked out)

      00:00:17.816 - U module5
      00:00:18.912 - At revision 48825
      00:00:19.228 - Fetching '<svn-url-for-module-6>' at -1 into '/mnt/workspace/8.30_Standard_Linux_build/build/module6'

      From what I can see the abnormalities are:

      • after the first checkout of module5/dir_4/file_100.xml there is a slight delay (approx 3 seconds)
      • At the end of the first checkout the line reporting at what revision we have reached is missing
      • the files checked out into module5/dir4 in the first checkout seems to be removed again.

      Without looking at the code or knowing anything I would guess that the subtask of checking out module-4 fails due to a network error or something but the main checkout task loses the error signal.

          [JENKINS-27263] Checkout of svn:externals fails silently and intermittently

          jens persson added a comment -

          To clarify:

          I think that the root course for the crash is external to the svn-plugin (most likely shaky network between the jenkins client and the svn server) but the main bug is that it fails silently.

          jens persson added a comment - To clarify: I think that the root course for the crash is external to the svn-plugin (most likely shaky network between the jenkins client and the svn server) but the main bug is that it fails silently.

          I'm experiencing the same issue with Subversion plugin version 2.10.5 and 2.11.1. During checkout/update of externals the process simply proceeds with the next external item. No "At revision <rev>" message shown.

          I don't experience a delay between the last checked out file and the next "<-- Got one external: ... " message.

          Both, Windows and Linux nodes are affected.

          Also, I don't believe this is a server or connection problem. The build server and the subversion server are on a local network. There have never been any network related problems. Using Jenkins, this is suddenly happening quite regularly. Using the command line client to update the working copy manually works without any problems. And this sort of problem has never occurred to any team member (using either TortoiseSVN client under Windows or SmartSVN or command line client under Linux). The server is a VisualSVN 3.6.3 (Subversion 1.9.5, Apache 2.2.34) on Windows Server 2012.

          There are no errors in the Jenkins log. But, there are errors in the Subversion Server log:

          Log Name: VisualSVNServer
          Source: VisualSVN Server 3.6
          Date: 8/4/2018 3:31:44 PM
          Event ID: 1001
          Task Category: Apache
          Level: Error
          Keywords: Classic
          User: >user<
          Computer: >server<
          Description:
          Unknown error [500, #106]
          [client xxx.xxx.xxx.xxx]
          Event Xml:
          <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
           <System>
           <Provider Name="VisualSVN Server 3.6" />
           <EventID Qualifiers="0">1001</EventID>
           <Level>2</Level>
           <Task>2</Task>
           <Keywords>0x80000000000000</Keywords>
           <TimeCreated SystemTime="2018-08-04T13:31:44.000000000Z" />
           <EventRecordID>55045</EventRecordID>
           <Channel>VisualSVNServer</Channel>
           <Computer>>server<</Computer>
           <Security UserID=">userid<" />
           </System>
           <EventData>
           <Data>Unknown error [500, #106]
          </Data>
           <Data>xxx.xxx.xxx.xxx</Data>
           </EventData>
          </Event>
          
          Log Name: VisualSVNServer
          Source: VisualSVN Server 3.6
          Date: 8/4/2018 3:31:44 PM
          Event ID: 1001
          Task Category: Apache
          Level: Error
          Keywords: Classic
          User: >user<
          Computer: >server<
          Description:
          A failure occurred while driving the update report editor [500, #106]
          [client xxx.xxx.xxx.xxx]
          Event Xml:
          <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
           <System>
           <Provider Name="VisualSVN Server 3.6" />
           <EventID Qualifiers="0">1001</EventID>
           <Level>2</Level>
           <Task>2</Task>
           <Keywords>0x80000000000000</Keywords>
           <TimeCreated SystemTime="2018-08-04T13:31:44.000000000Z" />
           <EventRecordID>55044</EventRecordID>
           <Channel>VisualSVNServer</Channel>
           <Computer>>server<</Computer>
           <Security UserID=">userid<" />
           </System>
           <EventData>
           <Data>A failure occurred while driving the update report editor [500, #106]
          </Data>
           <Data>xxx.xxx.xxx.xxx</Data>
           </EventData>
          </Event>
          
          Log Name: VisualSVNServer
          Source: VisualSVN Server 3.6
          Date: 8/4/2018 3:31:44 PM
          Event ID: 1001
          Task Category: Apache
          Level: Error
          Keywords: Classic
          User: >user<
          Computer: >server<
          Description:
          Provider encountered an error while streaming a REPORT response. [500, #0]
          [client xxx.xxx.xxx.xxx]
          Event Xml:
          <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
           <System>
           <Provider Name="VisualSVN Server 3.6" />
           <EventID Qualifiers="0">1001</EventID>
           <Level>2</Level>
           <Task>2</Task>
           <Keywords>0x80000000000000</Keywords>
           <TimeCreated SystemTime="2018-08-04T13:31:44.000000000Z" />
           <EventRecordID>55043</EventRecordID>
           <Channel>VisualSVNServer</Channel>
           <Computer>>server<</Computer>
           <Security UserID=">userid<" />
           </System>
           <EventData>
           <Data>Provider encountered an error while streaming a REPORT response. [500, #0]
          </Data>
           <Data>xxx.xxx.xxx.xxx</Data>
           </EventData>
          </Event>
          

          I'm not sure whether these errors are related to this issue or not. The errors occur during the check out of the external. But there are many more files checked out until the issue occurs and the next external gets processed (the last checked out file for the external has time stamp 16:07:35). As there are several parallel check outs running at that time the server errors might not be related. A quick web search shows that the errors might be related to client-side timeouts.

           

           

          Christoph Vogtländer added a comment - I'm experiencing the same issue with Subversion plugin version 2.10.5 and 2.11.1. During checkout/update of externals the process simply proceeds with the next external item. No "At revision <rev>" message shown. I don't experience a delay between the last checked out file and the next "<-- Got one external: ... " message. Both, Windows and Linux nodes are affected. Also, I don't believe this is a server or connection problem. The build server and the subversion server are on a local network. There have never been any network related problems. Using Jenkins, this is suddenly happening quite regularly. Using the command line client to update the working copy manually works without any problems. And this sort of problem has never occurred to any team member (using either TortoiseSVN client under Windows or SmartSVN or command line client under Linux). The server is a VisualSVN 3.6.3 (Subversion 1.9.5, Apache 2.2.34) on Windows Server 2012. There are no errors in the Jenkins log. But, there are errors in the Subversion Server log: Log Name: VisualSVNServer Source: VisualSVN Server 3.6 Date: 8/4/2018 3:31:44 PM Event ID: 1001 Task Category: Apache Level: Error Keywords: Classic User: >user< Computer: >server< Description: Unknown error [500, #106] [client xxx.xxx.xxx.xxx] Event Xml: <Event xmlns= "http: //schemas.microsoft.com/win/2004/08/events/event" > < System > <Provider Name= "VisualSVN Server 3.6" /> <EventID Qualifiers= "0" >1001</EventID> <Level>2</Level> <Task>2</Task> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime= "2018-08-04T13:31:44.000000000Z" /> <EventRecordID>55045</EventRecordID> <Channel>VisualSVNServer</Channel> <Computer>>server<</Computer> <Security UserID= ">userid<" /> </ System > <EventData> <Data>Unknown error [500, #106] </Data> <Data>xxx.xxx.xxx.xxx</Data> </EventData> </Event> Log Name: VisualSVNServer Source: VisualSVN Server 3.6 Date: 8/4/2018 3:31:44 PM Event ID: 1001 Task Category: Apache Level: Error Keywords: Classic User: >user< Computer: >server< Description: A failure occurred while driving the update report editor [500, #106] [client xxx.xxx.xxx.xxx] Event Xml: <Event xmlns= "http: //schemas.microsoft.com/win/2004/08/events/event" > < System > <Provider Name= "VisualSVN Server 3.6" /> <EventID Qualifiers= "0" >1001</EventID> <Level>2</Level> <Task>2</Task> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime= "2018-08-04T13:31:44.000000000Z" /> <EventRecordID>55044</EventRecordID> <Channel>VisualSVNServer</Channel> <Computer>>server<</Computer> <Security UserID= ">userid<" /> </ System > <EventData> <Data>A failure occurred while driving the update report editor [500, #106] </Data> <Data>xxx.xxx.xxx.xxx</Data> </EventData> </Event> Log Name: VisualSVNServer Source: VisualSVN Server 3.6 Date: 8/4/2018 3:31:44 PM Event ID: 1001 Task Category: Apache Level: Error Keywords: Classic User: >user< Computer: >server< Description: Provider encountered an error while streaming a REPORT response. [500, #0] [client xxx.xxx.xxx.xxx] Event Xml: <Event xmlns= "http: //schemas.microsoft.com/win/2004/08/events/event" > < System > <Provider Name= "VisualSVN Server 3.6" /> <EventID Qualifiers= "0" >1001</EventID> <Level>2</Level> <Task>2</Task> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime= "2018-08-04T13:31:44.000000000Z" /> <EventRecordID>55043</EventRecordID> <Channel>VisualSVNServer</Channel> <Computer>>server<</Computer> <Security UserID= ">userid<" /> </ System > <EventData> <Data>Provider encountered an error while streaming a REPORT response. [500, #0] </Data> <Data>xxx.xxx.xxx.xxx</Data> </EventData> </Event> I'm not sure whether these errors are related to this issue or not. The errors occur during the check out of the external. But there are many more files checked out until the issue occurs and the next external gets processed (the last checked out file for the external has time stamp 16:07:35). As there are several parallel check outs running at that time the server errors might not be related. A quick web search shows that the errors might be related to client-side timeouts.    

          Setting "cancelProcessOnExternalsFail: true" (checkout step) will fail the build. This is an example stack trace:

          [Checkout branch2] ERROR: Failed to update https://server/repo
          [Checkout branch2] org.tmatesoft.svn.core.SVNException: svn: E205011: Failed processing one or more externals definitions: <Qt>
          [Checkout branch2] 	at hudson.scm.subversion.SubversionUpdateEventHandler.handleEvent(SubversionUpdateEventHandler.java:145)
          [Checkout branch2] 	at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.handleEvent(SvnOperationRunner.java:67)
          [Checkout branch2] 	at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.handleEvent(SvnOperationRunner.java:72)
          [Checkout branch2] 	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.handleExternalsChange(SvnNgAbstractUpdate.java:383)
          [Checkout branch2] 	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.handleExternals(SvnNgAbstractUpdate.java:315)
          [Checkout branch2] 	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.updateInternal(SvnNgAbstractUpdate.java:279)
          [Checkout branch2] 	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.update(SvnNgAbstractUpdate.java:115)
          [Checkout branch2] 	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgUpdate.run(SvnNgUpdate.java:40)
          [Checkout branch2] 	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgUpdate.run(SvnNgUpdate.java:18)
          [Checkout branch2] 	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgOperationRunner.run(SvnNgOperationRunner.java:20)
          [Checkout branch2] 	at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:21)
          [Checkout branch2] 	at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1239)
          [Checkout branch2] 	at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
          [Checkout branch2] 	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:311)
          [Checkout branch2] 	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:291)
          [Checkout branch2] 	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:387)
          [Checkout branch2] 	at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:161)
          [Checkout branch2] 	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:168)
          [Checkout branch2] 	at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:1041)
          [Checkout branch2] 	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1017)
          [Checkout branch2] 	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:990)
          [Checkout branch2] 	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2913)
          [Checkout branch2] 	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
          [Checkout branch2] 	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
          [Checkout branch2] 	at hudson.remoting.Request$2.run(Request.java:369)
          [Checkout branch2] 	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
          [Checkout branch2] 	at java.util.concurrent.FutureTask.run(Unknown Source)
          [Checkout branch2] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
          [Checkout branch2] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          [Checkout branch2] 	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
          [Checkout branch2] 	at java.lang.Thread.run(Unknown Source)
          [Checkout branch2] ERROR: Subversion update failed
          

          So at least such an error won't get through unnoticed.

          Christoph Vogtländer added a comment - Setting "cancelProcessOnExternalsFail: true" (checkout step) will fail the build. This is an example stack trace: [Checkout branch2] ERROR: Failed to update https: //server/repo [Checkout branch2] org.tmatesoft.svn.core.SVNException: svn: E205011: Failed processing one or more externals definitions: <Qt> [Checkout branch2] at hudson.scm.subversion.SubversionUpdateEventHandler.handleEvent(SubversionUpdateEventHandler.java:145) [Checkout branch2] at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.handleEvent(SvnOperationRunner.java:67) [Checkout branch2] at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.handleEvent(SvnOperationRunner.java:72) [Checkout branch2] at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.handleExternalsChange(SvnNgAbstractUpdate.java:383) [Checkout branch2] at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.handleExternals(SvnNgAbstractUpdate.java:315) [Checkout branch2] at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.updateInternal(SvnNgAbstractUpdate.java:279) [Checkout branch2] at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.update(SvnNgAbstractUpdate.java:115) [Checkout branch2] at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgUpdate.run(SvnNgUpdate.java:40) [Checkout branch2] at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgUpdate.run(SvnNgUpdate.java:18) [Checkout branch2] at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgOperationRunner.run(SvnNgOperationRunner.java:20) [Checkout branch2] at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:21) [Checkout branch2] at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1239) [Checkout branch2] at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294) [Checkout branch2] at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:311) [Checkout branch2] at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:291) [Checkout branch2] at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:387) [Checkout branch2] at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:161) [Checkout branch2] at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:168) [Checkout branch2] at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:1041) [Checkout branch2] at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1017) [Checkout branch2] at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:990) [Checkout branch2] at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2913) [Checkout branch2] at hudson.remoting.UserRequest.perform(UserRequest.java:212) [Checkout branch2] at hudson.remoting.UserRequest.perform(UserRequest.java:54) [Checkout branch2] at hudson.remoting.Request$2.run(Request.java:369) [Checkout branch2] at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) [Checkout branch2] at java.util.concurrent.FutureTask.run(Unknown Source) [Checkout branch2] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [Checkout branch2] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [Checkout branch2] at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93) [Checkout branch2] at java.lang. Thread .run(Unknown Source) [Checkout branch2] ERROR: Subversion update failed So at least such an error won't get through unnoticed.

            Unassigned Unassigned
            mr_shark jens persson
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: