Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-25281

Mercurial plugin sometimes stalls when transferring bundle from master to slave

    • Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Icon: Major Major
    • mercurial-plugin
    • Jenkins 1.573, Mercurial plugin 1.50, Java Platform SE 6 U26, TortoiseHg 2.5.1
      Master: Windows 7 Enterprise 32-bit, Slaves: Windows 7 Professional 32 and 64-bit

      We have enabled 'Use Repository Caches' and 'Use Repository Sharing' for our Mercurial installation. After successfully creating the bundle, sometimes the xfer.hg file will stall at 0 bytes on the slave, particularly on the first attempt. Our repository is quite large, creating a 3GB bundle, so it's particularly frustrating when it stops at this point. I worked around this by manually creating a clone in the correct place on the slaves, but found occasional problems with the incremental bundles too. For example:

      Acquired slave node cache lock for node MYSLAVE.
      [579515DBB228E33DCBB4F032DF0E45CC959C39DD-omgsvr60-SWTest] $ hg --config auth.jenkins.prefix=* --config ******** --config ******** --config "auth.jenkins.schemes=http https" heads --template

      {node}\n --topo --closed
      [579515DBB228E33DCBB4F032DF0E45CC959C39DD-omgsvr60-SWTest] $ hg --config auth.jenkins.prefix=* --config ******** --config ******** --config "auth.jenkins.schemes=http https" heads --template {node}

      \n --topo --closed
      [579515DBB228E33DCBB4F032DF0E45CC959C39DD-myserver-myuser] $ hg --config auth.jenkins.prefix=* --config ******** --config ******** --config "auth.jenkins.schemes=http https" bundle --base c5a2e7c076fbfd7a32ad3cd0b73388c481d3fd27 --base 8d182a69f679fd2fb72f7d6b97503681cfe84052 --base ff4d5952ec39a97366e43a059c9fce7b60f5b366 --base 90c38b598baca76ed6a6eb3f8c270f4c47852ed7 --base d0ef3d70db45afbde1b609df3492499a774032b6 --base 78f1727e4fd2b3afc3a29abd9782483cbabd53b4 --base bbe7df97e6f2852a2f1fdbfe24896f45a7b067e7 --base 153e52c969fdf5017e15497a404d41ace9ec5d8c --base d6333be7a7917b7c46f0a4c5423eef26c032734d --base 397e7346ea5ca31c307106d4d369fd2b24d25b0f --base bbaaf47c9d646b35d3e8a1e6189c75eb194c2935 --base 8979c8d32cad2c7a45b424cdcbef14f08e0a290c --base ed2054ac69cc191dbc0985c137f35cca718eaa84 --base fd0807154ea9cf8490d3fb50170036f12a3b9ee9 --base 32f449e2d619d3f5d043cbe66b47eeaa23b78e7b --base 08af4ea4e2c351ade7430b133821b8da0aab841e --base 35149a612c00777d4c851a459271affdf05fd403 --base 6fd2d2f720b2d47bd30d2d6d0df5e0129f897a85 --base b174185b4d7c1bf922222b38332a068376d97318 --base 6d098ac5894004ab103da98d709b1bbcc045bbfe --base 538821c384d68e00a2377da5cbc708a36d3d1e29 --base 92d3469386e3ee76f254ac7f9d18fe2ef626ac17 --base 10d789a2ab40d1d4b101a30fa898b0cc4d29a971 --base d2f611447ebf4fa4d30fbeaf2de56cec06ddb4c6 --base 8c1ff350b064332591cabf8d6813f3c4842f362a --base 72b1ab5abf099df3b14c3198fd73813759b89af4 --base 37b6cb12de565f94ace2425a7f836f7f67ac16c1 --base 4e7d2805ba3b5dd78fc5caa68e56bc2c6b49af84 --base 9fa7582d7b46d73b140e1b683327dca0faa11d32 --base 07eebef65ce37ee977e11b609ea23ceeebb2f315 --base f902a366f3c010fa2fd8b2c63f46e7990adb913f --base b68a73582e262a95ea5c7799ea5d0a14be94e671 --base 20885f2d829345fc698c72fd7691b9e7158e19e4 --base 961482b343bf29e48da3db8bf76ac233cb255098 --base b9c2fe2e5b34e096edef7480f7c8807031298cb9 --base 58cb5e3288de308cec831c883d326e00405fac44 --base 5d1fa0946168dd9409a5449c52e0c43049511ea7 --base 9f906dff2aebd162266337440e6e9ece628114fe --base 692c66f7464ea2ec91418370e7e36c1755eeb3e1 --base 1d479c8f34442adad9c1c5b0fa5cdc9f9431a88e --base 52e39cb9d5d6f3a4ebbf8c9ca17f5afa6fffd902 --base 8dff9b1b103450101c7e23237fe13582f6c59c17 --base c0c7cfcc36241b3013cdb8bb560a8ad877f912eb --base 4db934c4a496173bbf0137be9d9d430a7389d908 --base 877ad8db6998bac2b6bb325b70e1b4af1513c4ae --base c282c12c7bc7284f67420eb51dff6b2fa621baac --base 67aa15a64cf208e1308b6d5645955918ac8895c8 --base 3cb31bb153e2026e437943afc2511b7adc77258a --base d48b9fb9c262aabc6601c2e8835cb7ca4ff3a49d --base bdabd261ab6ecfdee79e8079152e963ab692f259 --base 0986ca10ea907110b703128e53a11b678c0bc731 --base 67a5f6c772566a57ef9597a7c47d4320c5fe457a --base c29a4b97a0ea23450bf0138e39d27bc53841f1c3 --base 3b4251c11c2980c81b1624383a3545905a97572a --base ca7d879acf6147c44d72867504b93561c93c2895 --base 6caf901db17591229079485a34c6f51a73a2b1eb --base 6bbdd69793eb30462a2ba6125295a6f473b33e4d --base 2b791d8a805ead756993a003015f2b4f57b7e098 --base 2160fe71bfdb02b2b320a9544a6b52cba99832ca --base f0b07eab75e4ed283c99959995578f1976de4aa5 --base 1aceb40eabe5d05afb3f0d3f3288687a17addbed --base 9cb78d34a5a3b61e226980528c3b1bc4b072367f --base 4b8af702a84965fa3d692a02cf096dd6c1c8db1b --base b2b2483df53a83eb3180927af2cd3ba60d6ddb52 --base 480a1713acf57459e3700c2958ae7ec69e7ed66b --base 5e9061ab5147a136234c7a84bc543bc27526f049 --base 3a51a5e597756352e83b2a2c17df24a86bc23466 --base 9d2dfad58437a19a4a038636356d3b97bc420152 --base 2af30741f6850d08ca32d6ba9dd08782cabcf000 --base 33e462acbf1484dd33244c578a908c5338b1caa1 --base 723d84b6be6dc3566313502f925738396990ca99 --base 08dde1c6049fa6c60c43e58a0f5ea26de7e44721 --base f58ff1a3fc77d09ee89cf2dc117b67a01e9d7f88 --base bc6aed75b0d34e84b3d4e0c73a54afe990997f63 --base 7068904e7e12bc41e45480542fa7556debe0dcd3 --base 3acd3c9d4d58bc900470c47b5c6a9341672f5bef --base 1ce510217eff38f27bf5cc3bea4f655600596730 --base 41a3883e093f9a95e0c00ee0fdb622db2f9faad0 --base bda9e470cce253e70a3b8cafae9a4473506ae351 --base ef84d0494cc5a88fa12981738cf09f65066bca40 --base e2d69bf6685eecdc67b1616555e7fa7c1cf67844 --base 8adc47d90b5f8d6cca92ca280d38a347caf441f1 --base d538acd2a32b58631e3e3586d29b372ba4db2cac --base 9c3b8778d2edd1adfa7c0b5a0729a60f3096a569 --base e3ce4aa12ee1c691f8030b131b9268e69252ca59 --base 37679c57dde5a9040e63a52e1b22c65064f6c51a --base af97866868fffbcd1dedeb33de9dd93167f2f641 --base 0a4d2f73d79f03ed743640802f3bc177314184fb --base 927003b7f7633825f54c26217005363c4adb1844 --base b34bd3bf52e13d2afbd29344a52bc5e428be25bb --base 09b3bd8085d9fee0327357cbab7e94bb654fc799 --base 540b5939c1495ff218165dcb9466f49f0e2b3b85 --base 14ba0b0936b7f2c4541f17c1efdca9e617e783fb --base e3c2137a3861d6c83fe83fd829ea0ec5b5b2ea04 --base 795f92141914aa82ece95968029514bfdd40cacf --base 8b54e2e25a85a42f30b2ef8a6bbe573521da51db --base 835b5bc586ff5fcee551846825baab43410bc6ad --base bb02c1a7b54f42d77e05007980bd27dd2f977d1b --base f4c0d733a35f15780e9f77460c376b4774b0e79d --base 3a5a307a822e538c22e1264b59ca2dfc2ba04ee5 --base a9b4646843296115cd2e1a9452b4a8681438ffda --base 566fa833c250962e754200708decb0daf9f09fae --base 252e294196719f328d0531f642d7d2cf9b96d776 --base a2a17d7318c3b209c048d4e341fbbf6f9b0d7f60 --base 676733a8e1120e9e4c305b5a931720d92ba539eb --base f3c7cd79fbcb4b5ceb2c71a868325fb7b0c5f7d1 --base 720ab481fdcf95f1dac33c6669b043dea1c41bb2 --base 6f72db9c2feb101bd4cbdf173e47055d74a03d37 --base 9b02a1efe1c4451321ea4f7ef00d3bc125eff6d4 --base b01e05190594a0a201096b2fa8a093815775b712 --base 43fde762c8cd6e18273fd01c245d7ac011308980 --base d0b04c5ffad11dec5f5b82f48a125ad9d082e066 --base 43549ada152662ef23813aeb154458cd9c60eb61 --base e46f5c6672fe8ae89711d8fa774fd2e75f3795fa --base 4be34a677e7b806e2e66eafeef4bc5facd46e9f3 --base 0266671351efe7deccc4dcd77127f3b18b156d98 --base 41204097da2832c8ca70ace0ec4d744b4a989abe --base 44fc9a120d2bf11b919239f690b8232c1d944240 --base b5b04d56bc4f32ca2b5cccc1b7346f0fdc8e9e24 --base f7149a5265ae0668a010288383ae7b9eb7f6fe64 --base 577dac2a3192a915b253e011577ad143e33020f5 --base 55c49f2f9f62fa0cd2367e0e53ee3364df0c5eec --base ed5ed2238ea5152061af78c9dd32a6e755685c39 --base a3a24d8a82f11dde4c274bc79b60adb94e1675bb --base 62302397801ad4d4ea0f9412a09bd91d000dc564 --base 99f21a7d4c17146b6d797a2f5dd79e91097ba9a2 --base fb6f8a991d0eaed7a39d2ff81cc45ee7751d0e18 --base 0c292d3b1cec822f79e95508b15f238a8900a4f3 --base af489a5296d964536334098880dcdb4e72f24240 --base 2aa80c9c51fcc9e9268038b8aaef56393325a7bd --base ee39131fef7b86c10c7044fbfb08ba8f35b1e15a --base 2773681b377753c2fa726fc34209fc583cbab72b --base a47ed5d96e4106c146873751a90e619e9e2c5951 --base be4a745852a7780214e76bdddf8521b3f63c6053 --base 8463c1717228c5bc0e3064b00b1c783f5b8ea4a2 --base 873bd0cda302238cc223c59509ac371802ece8f3 --base 55ba01930cea350ad5134e323d349eb1a07b16ff --base 6b10bcd87f9ea24928a9e2dd4230f90f57d86fff --base 480fdbab11ad8c3278c35b9e74cf679706d69245 --base 1e28ab0eb2a922c5c04f84e278f9d11d92cf57f7 --base fe75dbfc2602cbc71a6faaa8270306f10c382ad6 --base c2c4befe7e6ca1a17fea25e5cb92f0fa64ea0147 --base b5b00b1234e178d22212fefb637f492814dd92a2 --base cb4859cb96438410cf8323cee4388690c9e0733f --base 0b2e89de78f358b38b9542c8b4163bd4198e8395 --base 410e802195360d6885c77a75835f2c8172e23431 --base b538bc7fb11595e7b722b13e0014df2414556bdf xfer-MYSLAVE.hg
      4 changesets found
      [579515DBB228E33DCBB4F032DF0E45CC959C39DD-myserver-myuser] $ hg --config auth.jenkins.prefix=* --config ******** --config ******** --config "auth.jenkins.schemes=http https" unbundle xfer.hg
      abort: stream ended unexpectedly (got 0 bytes, expected 6)
      ERROR: Failed to unbundle E:\Hudson\hgcache\579515DBB228E33DCBB4F032DF0E45CC959C39DD-myserver-myuser\xfer.hg
      Slave node cache lock released for node MYSLAVE.
      ERROR: Failed to use repository cache for http://myserver
      java.io.IOException: remote file operation failed: E:\Hudson\hgcache\579515DBB228E33DCBB4F032DF0E45CC959C39DD-myserver-myuser\xfer.hg at hudson.remoting.Channel@1d6faa5:MYSLAVE
      at hudson.FilePath.act(FilePath.java:918)
      at hudson.FilePath.act(FilePath.java:895)
      at hudson.FilePath.delete(FilePath.java:1329)
      at hudson.plugins.mercurial.Cache.repositoryCache(Cache.java:216)
      at hudson.plugins.mercurial.MercurialSCM.cachedSource(MercurialSCM.java:821)
      at hudson.plugins.mercurial.MercurialSCM.pull(MercurialSCM.java:391)
      at hudson.plugins.mercurial.MercurialSCM.update(MercurialSCM.java:598)
      at hudson.plugins.mercurial.MercurialSCM.checkout(MercurialSCM.java:481)
      at hudson.model.AbstractProject.checkout(AbstractProject.java:1252)
      at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:624)
      at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
      at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:530)
      at hudson.model.Run.execute(Run.java:1732)
      at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
      at hudson.model.ResourceController.execute(ResourceController.java:88)
      at hudson.model.Executor.run(Executor.java:234)
      Caused by: java.io.IOException: Unable to delete E:\Hudson\hgcache\579515DBB228E33DCBB4F032DF0E45CC959C39DD-myserver-myuser\xfer.hg
      at hudson.Util.deleteFile(Util.java:263)
      at hudson.FilePath$17.invoke(FilePath.java:1332)
      at hudson.FilePath$17.invoke(FilePath.java:1329)
      at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2492)
      at hudson.remoting.UserRequest.perform(UserRequest.java:118)
      at hudson.remoting.UserRequest.perform(UserRequest.java:48)
      at hudson.remoting.Request$2.run(Request.java:328)
      at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
      at java.util.concurrent.FutureTask.run(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      at hudson.remoting.Engine$1$1.run(Engine.java:63)
      at java.lang.Thread.run(Unknown Source)

      After this happened, the xfer.hg file was locked by Java, preventing subsequent updates until a manual restart of the slave process.

      Our Java version is rather old, so I'm going to try updating that, in case there's a problem in the mechanism by which it transfers the file over the network.

          [JENKINS-25281] Mercurial plugin sometimes stalls when transferring bundle from master to slave

          Kim Randell added a comment - - edited

          I've had one successful full 3GB bundle transfer since updating to JRE 7u72; unfortunately I won't get a chance to try another one for a little over a week. However, the delta bundle transfers will continue to run for that period and I'll report back on whether there are any failures.

          Kim Randell added a comment - - edited I've had one successful full 3GB bundle transfer since updating to JRE 7u72; unfortunately I won't get a chance to try another one for a little over a week. However, the delta bundle transfers will continue to run for that period and I'll report back on whether there are any failures.

          Jesse Glick added a comment -

          There are endless reasons why the master-slave communication channel can fail. Unless I see some indication to the contrary I doubt there is anything specific the Mercurial plugin could do to defend against this; probably a bug in Jenkins core. It seems that FilePath.copyTo(FilePath) created the target file (slave xfer.hg) but did not finish (or even start?) writing content to it, and left the file locked (which it definitely is not supposed to do).

          The workaround is just to not use the repository caching feature.

          Jesse Glick added a comment - There are endless reasons why the master-slave communication channel can fail. Unless I see some indication to the contrary I doubt there is anything specific the Mercurial plugin could do to defend against this; probably a bug in Jenkins core. It seems that FilePath.copyTo(FilePath) created the target file (slave xfer.hg ) but did not finish (or even start?) writing content to it, and left the file locked (which it definitely is not supposed to do). The workaround is just to not use the repository caching feature.

          Kim Randell added a comment -

          We've had it happen one more time on a delta transfer. If/when it happens again I'll try to inspect the thread dumps on the master and slave and see if I can spot where FilePath.copyTo is getting stuck.

          Kim Randell added a comment - We've had it happen one more time on a delta transfer. If/when it happens again I'll try to inspect the thread dumps on the master and slave and see if I can spot where FilePath.copyTo is getting stuck.

            jglick Jesse Glick
            krandell Kim Randell
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: