• Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • subversion-plugin
    • None
    • Hudson 1.356
      Hudson Subversion plugin 1.17
      Maven 2 project plugin 1.356

      Subversion 1.6.5,
      apr 1.3.8
      apr-utils 1.3.9
      neon 0.28.6
      OpenSSL 0.9.8k 25 Mar 2009
      zlib 1.2.3

      I have a system with about 70 jobs.
      The jobs are configured to poll svn every 5 mins and a nightly build is also scheduled for all the jobs.
      During the day changes in svn are properly detected and the builds triggered.

      But all the nightly builds fail with the error below:

      Building on master
      Updating http://svn:81/svn/...
      ERROR: Failed to update http://svn:81/svn/...
      org.tmatesoft.svn.core.SVNException: svn: OPTIONS of '/svn/...': 503 Service Unavailable (http://svn:81)
      svn: OPTIONS request failed on '/svn/Services/Valeurs2J2EE/branches/maintenance'
                      at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:103)
                      at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:87)
                      at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:616)
                      at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:273)
                      at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:261)
                      at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.exchangeCapabilities(DAVConnection.java:516)
                      at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.open(DAVConnection.java:98)
                      at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.openConnection(DAVRepository.java:1001)
                      at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getRepositoryUUID(DAVRepository.java:146)
                      at org.tmatesoft.svn.core.wc.SVNBasicClient.createRepository(SVNBasicClient.java:342)
                      at org.tmatesoft.svn.core.wc.SVNBasicClient.createRepository(SVNBasicClient.java:330)
                      at org.tmatesoft.svn.core.wc.SVNUpdateClient.update(SVNUpdateClient.java:535)
                      at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:401)
                      at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:699)
                      at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:660)
                      at hudson.FilePath.act(FilePath.java:747)
                      at hudson.FilePath.act(FilePath.java:729)
                      at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:653)
                      at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:601)
                      at hudson.model.AbstractProject.checkout(AbstractProject.java:1044)
                      at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479)
                      at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411)
                      at hudson.model.Run.run(Run.java:1244)
                      at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:306)
                      at hudson.model.ResourceController.execute(ResourceController.java:88)
                      at hudson.model.Executor.run(Executor.java:122)
      

      What I see is that the jobs are starting and failing a few seconds apart.

          [JENKINS-6481] svn 503 Service Unavailable error

          myannou added a comment -

          We also get very often this error during our nightly builds.
          We have the same conf except we run on windows OS

          I think there are 2 issues:

          • The 503 error itself (not sure if it doesn't come from SVN)
          • The fact that once the error happens then we cannot recover from the situation until we restart the hudson server.

          Any idea ?

          myannou added a comment - We also get very often this error during our nightly builds. We have the same conf except we run on windows OS I think there are 2 issues: The 503 error itself (not sure if it doesn't come from SVN) The fact that once the error happens then we cannot recover from the situation until we restart the hudson server. Any idea ?

          jgon added a comment -

          I have recently tested doing a svn update at regular intervals (10 mins) during our nightly builds using svn in command line without using Hudson at all.

          The outcome is that svn updates via the command line work fine while the updates via Hudson fail with 503 Service Unavailable.

          Additionally I have disabled building maven sites during the nightly builds to reduce access to svn.
          It has been working for less than a week. Afterwards I got 503 Service Unavailable even during the day and needed to restart Hudson.

          The machine where the builds run is rebooted weekly. After the reboot the builds work fine for a few days but end up failing.

          It looks like Hudson is leaking some resource overtime when accessing svn with a certain amount of jobs (about 70 in my case).

          I did run Hudson for about 2 months on 20 projects, including maven site building, without any problems.

          Looks like Hudson svn integration doesn't scale well.

          jgon added a comment - I have recently tested doing a svn update at regular intervals (10 mins) during our nightly builds using svn in command line without using Hudson at all. The outcome is that svn updates via the command line work fine while the updates via Hudson fail with 503 Service Unavailable. Additionally I have disabled building maven sites during the nightly builds to reduce access to svn. It has been working for less than a week. Afterwards I got 503 Service Unavailable even during the day and needed to restart Hudson. The machine where the builds run is rebooted weekly. After the reboot the builds work fine for a few days but end up failing. It looks like Hudson is leaking some resource overtime when accessing svn with a certain amount of jobs (about 70 in my case). I did run Hudson for about 2 months on 20 projects, including maven site building, without any problems. Looks like Hudson svn integration doesn't scale well.

          jgon added a comment -

          An update at the Hudson 503 service unavailable error.

          Sniffing the network traffic reveals that Hudson contacts in my case a company's internet proxy machine instead of the subversion server when getting the 503 service unavailable error.

          When jobs succeed, there is a call from the Hudson machine to the DNS asking for the address of the subversion server.
          But when jobs fail, there is call from Hudson to the DNS with the address of the company's internet proxy - I still haven't figured out why/how this switch takes place.

          I did another test:

          I replaced the subversion server DNS alias by its ip address and all jobs ran successfully - no more "503 service unavailable" errors.

          The latest test I am running is configuring the subversion server ip address in windows's "host" file to prevent access to the DNS server.

          Let's see what happens next -

          I'll keep you updated.

          jgon added a comment - An update at the Hudson 503 service unavailable error. Sniffing the network traffic reveals that Hudson contacts in my case a company's internet proxy machine instead of the subversion server when getting the 503 service unavailable error. When jobs succeed, there is a call from the Hudson machine to the DNS asking for the address of the subversion server. But when jobs fail, there is call from Hudson to the DNS with the address of the company's internet proxy - I still haven't figured out why/how this switch takes place. I did another test: I replaced the subversion server DNS alias by its ip address and all jobs ran successfully - no more "503 service unavailable" errors. The latest test I am running is configuring the subversion server ip address in windows's "host" file to prevent access to the DNS server. Let's see what happens next - I'll keep you updated.

          jgon added a comment -

          Configuring the subversion server ip address in the windows host file did not prevent the nightly builds from failing. After running a few jobs successfully we got 503 service unavailable errors for all the other jobs.

          The workaround has been to reconfigure all jobs to use the ip address of the svn server.

          Despite removing all references to the company's proxy in Hudson's configuration, the jobs failing were still communicating with the company's proxy.

          Looking at Hudson's settings I noticed that the Java system property http.proxyHost was set to the company's proxy and I am assuming that's were the company's proxy is picked up.

          My question here is what uses the http.proxyHost property: Hudson or the Hudson's subversion plugin?

          There is certainly an issue here because when something goes wrong either Hudson or the subversion plugin don't recover anymore.

          BTW, I forgot to mention that we are running Hudson 1.367.

          jgon added a comment - Configuring the subversion server ip address in the windows host file did not prevent the nightly builds from failing. After running a few jobs successfully we got 503 service unavailable errors for all the other jobs. The workaround has been to reconfigure all jobs to use the ip address of the svn server. Despite removing all references to the company's proxy in Hudson's configuration, the jobs failing were still communicating with the company's proxy. Looking at Hudson's settings I noticed that the Java system property http.proxyHost was set to the company's proxy and I am assuming that's were the company's proxy is picked up. My question here is what uses the http.proxyHost property: Hudson or the Hudson's subversion plugin? There is certainly an issue here because when something goes wrong either Hudson or the subversion plugin don't recover anymore. BTW, I forgot to mention that we are running Hudson 1.367.

            Unassigned Unassigned
            jgon jgon
            Votes:
            2 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: