Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-2351

Moving Hudson to new server out of sync error

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Reopened (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: subversion-plugin
    • Labels:
      None
    • Environment:
      Platform: All, OS: All
    • Similar Issues:

      Description

      Hi,

      I moved my jboss (has hudson deployed) to some other linux server. and also
      .hudson folder to this linux machine.

      when I start the hudson on new server and do a build, my build is not getting
      latest svn changes

      instead am getting the following message in the console output

      WARNING: clock of the subversion server appears to be out of sync. This can
      result in inconsistent check out behavior.

      WARNING: clock of the subversion server appears to be out of sync. This can
      result in inconsistent check out behavior.

      any 1 can help to get back my build on track ?

        Attachments

          Activity

          chamarts chamarts created issue -
          Hide
          steveleach steveleach added a comment -

          We are also seeing this. We support multiple Hudson servers connecting to
          multiple SVN repositories; all the Hudson servers are hosted on Red Hat linux
          virtual machines. The Hudson servers are time sync'd with the SVN servers.

          One effect is that where projects poll for changes they may occasionally miss an
          opportunity to build, so it can be up to twice the polling interval between the
          change to SVN and the build starting. When using "build now", we have to wait
          for a minute or so before pushing the button.

          I would like to understand how Hudson's SVN polling detects changes? Is the
          problem likely to be in Hudson's polling code, or the underlying SVN access
          library? Why are the dates on the servers even relevant? Can Hudson not detect
          changes based on revision numbers alone?

          Show
          steveleach steveleach added a comment - We are also seeing this. We support multiple Hudson servers connecting to multiple SVN repositories; all the Hudson servers are hosted on Red Hat linux virtual machines. The Hudson servers are time sync'd with the SVN servers. One effect is that where projects poll for changes they may occasionally miss an opportunity to build, so it can be up to twice the polling interval between the change to SVN and the build starting. When using "build now", we have to wait for a minute or so before pushing the button. I would like to understand how Hudson's SVN polling detects changes? Is the problem likely to be in Hudson's polling code, or the underlying SVN access library? Why are the dates on the servers even relevant? Can Hudson not detect changes based on revision numbers alone?
          Hide
          tomsquest tomsquest added a comment -

          Same error with SVN here :
          "WARNING: clock of the subversion server appears to be out of sync. This can
          result in inconsistent check out behavior."

          We recently change our source repository (svn dump && restore && new checkout)
          and then hudson started to produce the above error message.

          Disabling the use of update (svn will do a full checkout) did not solve the problem.

          So currently, whenever we do a build (manually or by trigger), Hudson is "late".

          Test case :

          • Hudson is setup to receive an event from SVN to launch the build (trigger)
          • I do a commit : SVN message : "Committed revision 20946"
          • Hudson fires a build
          • Hudson console output Updating[...] At revision 20945
          • Hudson then prints : WARNING: clock of the subversion server appears to be out
            of sync. This can result in inconsistent check out behavior.

          Of course the server (SVN, hudson) are synchronized...

          Show
          tomsquest tomsquest added a comment - Same error with SVN here : "WARNING: clock of the subversion server appears to be out of sync. This can result in inconsistent check out behavior." We recently change our source repository (svn dump && restore && new checkout) and then hudson started to produce the above error message. Disabling the use of update (svn will do a full checkout) did not solve the problem. So currently, whenever we do a build (manually or by trigger), Hudson is "late". Test case : Hudson is setup to receive an event from SVN to launch the build (trigger) I do a commit : SVN message : "Committed revision 20946" Hudson fires a build Hudson console output Updating [...] At revision 20945 Hudson then prints : WARNING: clock of the subversion server appears to be out of sync. This can result in inconsistent check out behavior. Of course the server (SVN, hudson) are synchronized...
          Hide
          davidedalto davidedalto added a comment -

          Hi,
          I took a look at the code and it seems that hudson compares its date with the
          date on the SVN server. If the date on hudson is after the date on the SVN,
          hudson doesn't checkout the project.

          Is there any reason for this? I would leave the warning message but I think that
          hudson should always checkout the last version.

          I am still investigating on this anyway.

          Show
          davidedalto davidedalto added a comment - Hi, I took a look at the code and it seems that hudson compares its date with the date on the SVN server. If the date on hudson is after the date on the SVN, hudson doesn't checkout the project. Is there any reason for this? I would leave the warning message but I think that hudson should always checkout the last version. I am still investigating on this anyway.
          Hide
          mdp mdp added a comment -

          @davidedalto
          From what I see the comparison is only used to print the warning and does not influence the checkout/update.

          The basic reason for the polling problems and the warning seems to be that in CheckOutTask#getRevision the svn plugin defaults to selecting a revision based on the timestamp of the build. This means that when the clock on the SVN server is ahead of the clock on the Hudson machine an older revision may be selected.
          Why it doesn't default to HEAD instead, I'm not sure, so I assume there must be a good reason. (Is the build timestamp used determined when the build is queued or when it is started?)
          Maybe the problem could be diminished with the help of a post-commit hook if the plugin did use the rev parameter shown in the wiki example, but apparently it is unused now.

          Show
          mdp mdp added a comment - @davidedalto From what I see the comparison is only used to print the warning and does not influence the checkout/update. The basic reason for the polling problems and the warning seems to be that in CheckOutTask#getRevision the svn plugin defaults to selecting a revision based on the timestamp of the build. This means that when the clock on the SVN server is ahead of the clock on the Hudson machine an older revision may be selected. Why it doesn't default to HEAD instead, I'm not sure, so I assume there must be a good reason. (Is the build timestamp used determined when the build is queued or when it is started?) Maybe the problem could be diminished with the help of a post-commit hook if the plugin did use the rev parameter shown in the wiki example , but apparently it is unused now.
          Hide
          lukewpatterson lukewpatterson added a comment -

          lost some hours to this, couple of simple things that would help, that wouldn't require a full fix:

          • show warning message even if build is started manually
          • include timezone info on main page (lower right corner of screen), right now only time/date is shown
          Show
          lukewpatterson lukewpatterson added a comment - lost some hours to this, couple of simple things that would help, that wouldn't require a full fix: show warning message even if build is started manually include timezone info on main page (lower right corner of screen), right now only time/date is shown
          Hide
          kohsuke Kohsuke Kawaguchi added a comment -

          What the Subversion plugin as of 1.37 (and it's been like this for a while) is that we ask the Subversion repository to tell us when the last revision was committed, and if that's coming from the future timestamp than the time of the check, then issue this warning.

          So if you are seeing this, your Subversion server is really out of sync with the slave where the build is happening.

          To make builds consistent (especially for matrix projects), we do timestamped check out, which is why we can't just check out HEAD.

          Show
          kohsuke Kohsuke Kawaguchi added a comment - What the Subversion plugin as of 1.37 (and it's been like this for a while) is that we ask the Subversion repository to tell us when the last revision was committed, and if that's coming from the future timestamp than the time of the check, then issue this warning. So if you are seeing this, your Subversion server is really out of sync with the slave where the build is happening. To make builds consistent (especially for matrix projects), we do timestamped check out, which is why we can't just check out HEAD.
          kohsuke Kohsuke Kawaguchi made changes -
          Field Original Value New Value
          Assignee Kohsuke Kawaguchi [ kohsuke ]
          Resolution Not A Defect [ 7 ]
          Status Open [ 1 ] Resolved [ 5 ]
          Hide
          acocarla Alecsandru Cocarla added a comment -

          Hi,

          We have this situation where the environment in which the Jenkins server runs, although it has the same timezone as the SVN server, it is intentionally out of synch (long story, but this is how it is...)
          Because of this issue, we can only build with @HEAD added to the SVN repo URL, but, as you mentioned, this means distributed building is no longer safe.
          My suggestion is to use a revision number instead of a timestamp to uniquely identify the build's source tree.

          Is this doable?

          Thank you

          Show
          acocarla Alecsandru Cocarla added a comment - Hi, We have this situation where the environment in which the Jenkins server runs, although it has the same timezone as the SVN server, it is intentionally out of synch (long story, but this is how it is...) Because of this issue, we can only build with @HEAD added to the SVN repo URL, but, as you mentioned, this means distributed building is no longer safe. My suggestion is to use a revision number instead of a timestamp to uniquely identify the build's source tree. Is this doable? Thank you
          acocarla Alecsandru Cocarla made changes -
          Resolution Not A Defect [ 7 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Hide
          ilynaf ily naf added a comment -

          The same problem here. Once a commit is made during the time when SVN server is out of sync (let's say one month ahead the current time), there is no way for you to have the job check out the latest rev again(it's always stuck at some rev ahead of that "guilty" commit). Although I've tried to clean up the working copy & check out again. The only workaround is added @HEAD to the repo URL. This is really frustrating

          Show
          ilynaf ily naf added a comment - The same problem here. Once a commit is made during the time when SVN server is out of sync (let's say one month ahead the current time), there is no way for you to have the job check out the latest rev again(it's always stuck at some rev ahead of that "guilty" commit). Although I've tried to clean up the working copy & check out again. The only workaround is added @HEAD to the repo URL. This is really frustrating
          rtyler R. Tyler Croy made changes -
          Workflow JNJira [ 132424 ] JNJira + In-Review [ 185897 ]

            People

            Assignee:
            kohsuke Kohsuke Kawaguchi
            Reporter:
            chamarts chamarts
            Votes:
            11 Vote for this issue
            Watchers:
            7 Start watching this issue

              Dates

              Created:
              Updated: