Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-25281

Mercurial plugin sometimes stalls when transferring bundle from master to slave

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Icon: Major Major
    • mercurial-plugin
    • Jenkins 1.573, Mercurial plugin 1.50, Java Platform SE 6 U26, TortoiseHg 2.5.1
      Master: Windows 7 Enterprise 32-bit, Slaves: Windows 7 Professional 32 and 64-bit

      We have enabled 'Use Repository Caches' and 'Use Repository Sharing' for our Mercurial installation. After successfully creating the bundle, sometimes the xfer.hg file will stall at 0 bytes on the slave, particularly on the first attempt. Our repository is quite large, creating a 3GB bundle, so it's particularly frustrating when it stops at this point. I worked around this by manually creating a clone in the correct place on the slaves, but found occasional problems with the incremental bundles too. For example:

      Acquired slave node cache lock for node MYSLAVE.
      [579515DBB228E33DCBB4F032DF0E45CC959C39DD-omgsvr60-SWTest] $ hg --config auth.jenkins.prefix=* --config ******** --config ******** --config "auth.jenkins.schemes=http https" heads --template

      {node}\n --topo --closed
      [579515DBB228E33DCBB4F032DF0E45CC959C39DD-omgsvr60-SWTest] $ hg --config auth.jenkins.prefix=* --config ******** --config ******** --config "auth.jenkins.schemes=http https" heads --template {node}

      \n --topo --closed
      [579515DBB228E33DCBB4F032DF0E45CC959C39DD-myserver-myuser] $ hg --config auth.jenkins.prefix=* --config ******** --config ******** --config "auth.jenkins.schemes=http https" bundle --base c5a2e7c076fbfd7a32ad3cd0b73388c481d3fd27 --base 8d182a69f679fd2fb72f7d6b97503681cfe84052 --base ff4d5952ec39a97366e43a059c9fce7b60f5b366 --base 90c38b598baca76ed6a6eb3f8c270f4c47852ed7 --base d0ef3d70db45afbde1b609df3492499a774032b6 --base 78f1727e4fd2b3afc3a29abd9782483cbabd53b4 --base bbe7df97e6f2852a2f1fdbfe24896f45a7b067e7 --base 153e52c969fdf5017e15497a404d41ace9ec5d8c --base d6333be7a7917b7c46f0a4c5423eef26c032734d --base 397e7346ea5ca31c307106d4d369fd2b24d25b0f --base bbaaf47c9d646b35d3e8a1e6189c75eb194c2935 --base 8979c8d32cad2c7a45b424cdcbef14f08e0a290c --base ed2054ac69cc191dbc0985c137f35cca718eaa84 --base fd0807154ea9cf8490d3fb50170036f12a3b9ee9 --base 32f449e2d619d3f5d043cbe66b47eeaa23b78e7b --base 08af4ea4e2c351ade7430b133821b8da0aab841e --base 35149a612c00777d4c851a459271affdf05fd403 --base 6fd2d2f720b2d47bd30d2d6d0df5e0129f897a85 --base b174185b4d7c1bf922222b38332a068376d97318 --base 6d098ac5894004ab103da98d709b1bbcc045bbfe --base 538821c384d68e00a2377da5cbc708a36d3d1e29 --base 92d3469386e3ee76f254ac7f9d18fe2ef626ac17 --base 10d789a2ab40d1d4b101a30fa898b0cc4d29a971 --base d2f611447ebf4fa4d30fbeaf2de56cec06ddb4c6 --base 8c1ff350b064332591cabf8d6813f3c4842f362a --base 72b1ab5abf099df3b14c3198fd73813759b89af4 --base 37b6cb12de565f94ace2425a7f836f7f67ac16c1 --base 4e7d2805ba3b5dd78fc5caa68e56bc2c6b49af84 --base 9fa7582d7b46d73b140e1b683327dca0faa11d32 --base 07eebef65ce37ee977e11b609ea23ceeebb2f315 --base f902a366f3c010fa2fd8b2c63f46e7990adb913f --base b68a73582e262a95ea5c7799ea5d0a14be94e671 --base 20885f2d829345fc698c72fd7691b9e7158e19e4 --base 961482b343bf29e48da3db8bf76ac233cb255098 --base b9c2fe2e5b34e096edef7480f7c8807031298cb9 --base 58cb5e3288de308cec831c883d326e00405fac44 --base 5d1fa0946168dd9409a5449c52e0c43049511ea7 --base 9f906dff2aebd162266337440e6e9ece628114fe --base 692c66f7464ea2ec91418370e7e36c1755eeb3e1 --base 1d479c8f34442adad9c1c5b0fa5cdc9f9431a88e --base 52e39cb9d5d6f3a4ebbf8c9ca17f5afa6fffd902 --base 8dff9b1b103450101c7e23237fe13582f6c59c17 --base c0c7cfcc36241b3013cdb8bb560a8ad877f912eb --base 4db934c4a496173bbf0137be9d9d430a7389d908 --base 877ad8db6998bac2b6bb325b70e1b4af1513c4ae --base c282c12c7bc7284f67420eb51dff6b2fa621baac --base 67aa15a64cf208e1308b6d5645955918ac8895c8 --base 3cb31bb153e2026e437943afc2511b7adc77258a --base d48b9fb9c262aabc6601c2e8835cb7ca4ff3a49d --base bdabd261ab6ecfdee79e8079152e963ab692f259 --base 0986ca10ea907110b703128e53a11b678c0bc731 --base 67a5f6c772566a57ef9597a7c47d4320c5fe457a --base c29a4b97a0ea23450bf0138e39d27bc53841f1c3 --base 3b4251c11c2980c81b1624383a3545905a97572a --base ca7d879acf6147c44d72867504b93561c93c2895 --base 6caf901db17591229079485a34c6f51a73a2b1eb --base 6bbdd69793eb30462a2ba6125295a6f473b33e4d --base 2b791d8a805ead756993a003015f2b4f57b7e098 --base 2160fe71bfdb02b2b320a9544a6b52cba99832ca --base f0b07eab75e4ed283c99959995578f1976de4aa5 --base 1aceb40eabe5d05afb3f0d3f3288687a17addbed --base 9cb78d34a5a3b61e226980528c3b1bc4b072367f --base 4b8af702a84965fa3d692a02cf096dd6c1c8db1b --base b2b2483df53a83eb3180927af2cd3ba60d6ddb52 --base 480a1713acf57459e3700c2958ae7ec69e7ed66b --base 5e9061ab5147a136234c7a84bc543bc27526f049 --base 3a51a5e597756352e83b2a2c17df24a86bc23466 --base 9d2dfad58437a19a4a038636356d3b97bc420152 --base 2af30741f6850d08ca32d6ba9dd08782cabcf000 --base 33e462acbf1484dd33244c578a908c5338b1caa1 --base 723d84b6be6dc3566313502f925738396990ca99 --base 08dde1c6049fa6c60c43e58a0f5ea26de7e44721 --base f58ff1a3fc77d09ee89cf2dc117b67a01e9d7f88 --base bc6aed75b0d34e84b3d4e0c73a54afe990997f63 --base 7068904e7e12bc41e45480542fa7556debe0dcd3 --base 3acd3c9d4d58bc900470c47b5c6a9341672f5bef --base 1ce510217eff38f27bf5cc3bea4f655600596730 --base 41a3883e093f9a95e0c00ee0fdb622db2f9faad0 --base bda9e470cce253e70a3b8cafae9a4473506ae351 --base ef84d0494cc5a88fa12981738cf09f65066bca40 --base e2d69bf6685eecdc67b1616555e7fa7c1cf67844 --base 8adc47d90b5f8d6cca92ca280d38a347caf441f1 --base d538acd2a32b58631e3e3586d29b372ba4db2cac --base 9c3b8778d2edd1adfa7c0b5a0729a60f3096a569 --base e3ce4aa12ee1c691f8030b131b9268e69252ca59 --base 37679c57dde5a9040e63a52e1b22c65064f6c51a --base af97866868fffbcd1dedeb33de9dd93167f2f641 --base 0a4d2f73d79f03ed743640802f3bc177314184fb --base 927003b7f7633825f54c26217005363c4adb1844 --base b34bd3bf52e13d2afbd29344a52bc5e428be25bb --base 09b3bd8085d9fee0327357cbab7e94bb654fc799 --base 540b5939c1495ff218165dcb9466f49f0e2b3b85 --base 14ba0b0936b7f2c4541f17c1efdca9e617e783fb --base e3c2137a3861d6c83fe83fd829ea0ec5b5b2ea04 --base 795f92141914aa82ece95968029514bfdd40cacf --base 8b54e2e25a85a42f30b2ef8a6bbe573521da51db --base 835b5bc586ff5fcee551846825baab43410bc6ad --base bb02c1a7b54f42d77e05007980bd27dd2f977d1b --base f4c0d733a35f15780e9f77460c376b4774b0e79d --base 3a5a307a822e538c22e1264b59ca2dfc2ba04ee5 --base a9b4646843296115cd2e1a9452b4a8681438ffda --base 566fa833c250962e754200708decb0daf9f09fae --base 252e294196719f328d0531f642d7d2cf9b96d776 --base a2a17d7318c3b209c048d4e341fbbf6f9b0d7f60 --base 676733a8e1120e9e4c305b5a931720d92ba539eb --base f3c7cd79fbcb4b5ceb2c71a868325fb7b0c5f7d1 --base 720ab481fdcf95f1dac33c6669b043dea1c41bb2 --base 6f72db9c2feb101bd4cbdf173e47055d74a03d37 --base 9b02a1efe1c4451321ea4f7ef00d3bc125eff6d4 --base b01e05190594a0a201096b2fa8a093815775b712 --base 43fde762c8cd6e18273fd01c245d7ac011308980 --base d0b04c5ffad11dec5f5b82f48a125ad9d082e066 --base 43549ada152662ef23813aeb154458cd9c60eb61 --base e46f5c6672fe8ae89711d8fa774fd2e75f3795fa --base 4be34a677e7b806e2e66eafeef4bc5facd46e9f3 --base 0266671351efe7deccc4dcd77127f3b18b156d98 --base 41204097da2832c8ca70ace0ec4d744b4a989abe --base 44fc9a120d2bf11b919239f690b8232c1d944240 --base b5b04d56bc4f32ca2b5cccc1b7346f0fdc8e9e24 --base f7149a5265ae0668a010288383ae7b9eb7f6fe64 --base 577dac2a3192a915b253e011577ad143e33020f5 --base 55c49f2f9f62fa0cd2367e0e53ee3364df0c5eec --base ed5ed2238ea5152061af78c9dd32a6e755685c39 --base a3a24d8a82f11dde4c274bc79b60adb94e1675bb --base 62302397801ad4d4ea0f9412a09bd91d000dc564 --base 99f21a7d4c17146b6d797a2f5dd79e91097ba9a2 --base fb6f8a991d0eaed7a39d2ff81cc45ee7751d0e18 --base 0c292d3b1cec822f79e95508b15f238a8900a4f3 --base af489a5296d964536334098880dcdb4e72f24240 --base 2aa80c9c51fcc9e9268038b8aaef56393325a7bd --base ee39131fef7b86c10c7044fbfb08ba8f35b1e15a --base 2773681b377753c2fa726fc34209fc583cbab72b --base a47ed5d96e4106c146873751a90e619e9e2c5951 --base be4a745852a7780214e76bdddf8521b3f63c6053 --base 8463c1717228c5bc0e3064b00b1c783f5b8ea4a2 --base 873bd0cda302238cc223c59509ac371802ece8f3 --base 55ba01930cea350ad5134e323d349eb1a07b16ff --base 6b10bcd87f9ea24928a9e2dd4230f90f57d86fff --base 480fdbab11ad8c3278c35b9e74cf679706d69245 --base 1e28ab0eb2a922c5c04f84e278f9d11d92cf57f7 --base fe75dbfc2602cbc71a6faaa8270306f10c382ad6 --base c2c4befe7e6ca1a17fea25e5cb92f0fa64ea0147 --base b5b00b1234e178d22212fefb637f492814dd92a2 --base cb4859cb96438410cf8323cee4388690c9e0733f --base 0b2e89de78f358b38b9542c8b4163bd4198e8395 --base 410e802195360d6885c77a75835f2c8172e23431 --base b538bc7fb11595e7b722b13e0014df2414556bdf xfer-MYSLAVE.hg
      4 changesets found
      [579515DBB228E33DCBB4F032DF0E45CC959C39DD-myserver-myuser] $ hg --config auth.jenkins.prefix=* --config ******** --config ******** --config "auth.jenkins.schemes=http https" unbundle xfer.hg
      abort: stream ended unexpectedly (got 0 bytes, expected 6)
      ERROR: Failed to unbundle E:\Hudson\hgcache\579515DBB228E33DCBB4F032DF0E45CC959C39DD-myserver-myuser\xfer.hg
      Slave node cache lock released for node MYSLAVE.
      ERROR: Failed to use repository cache for http://myserver
      java.io.IOException: remote file operation failed: E:\Hudson\hgcache\579515DBB228E33DCBB4F032DF0E45CC959C39DD-myserver-myuser\xfer.hg at hudson.remoting.Channel@1d6faa5:MYSLAVE
      at hudson.FilePath.act(FilePath.java:918)
      at hudson.FilePath.act(FilePath.java:895)
      at hudson.FilePath.delete(FilePath.java:1329)
      at hudson.plugins.mercurial.Cache.repositoryCache(Cache.java:216)
      at hudson.plugins.mercurial.MercurialSCM.cachedSource(MercurialSCM.java:821)
      at hudson.plugins.mercurial.MercurialSCM.pull(MercurialSCM.java:391)
      at hudson.plugins.mercurial.MercurialSCM.update(MercurialSCM.java:598)
      at hudson.plugins.mercurial.MercurialSCM.checkout(MercurialSCM.java:481)
      at hudson.model.AbstractProject.checkout(AbstractProject.java:1252)
      at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:624)
      at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
      at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:530)
      at hudson.model.Run.execute(Run.java:1732)
      at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
      at hudson.model.ResourceController.execute(ResourceController.java:88)
      at hudson.model.Executor.run(Executor.java:234)
      Caused by: java.io.IOException: Unable to delete E:\Hudson\hgcache\579515DBB228E33DCBB4F032DF0E45CC959C39DD-myserver-myuser\xfer.hg
      at hudson.Util.deleteFile(Util.java:263)
      at hudson.FilePath$17.invoke(FilePath.java:1332)
      at hudson.FilePath$17.invoke(FilePath.java:1329)
      at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2492)
      at hudson.remoting.UserRequest.perform(UserRequest.java:118)
      at hudson.remoting.UserRequest.perform(UserRequest.java:48)
      at hudson.remoting.Request$2.run(Request.java:328)
      at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
      at java.util.concurrent.FutureTask.run(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      at hudson.remoting.Engine$1$1.run(Engine.java:63)
      at java.lang.Thread.run(Unknown Source)

      After this happened, the xfer.hg file was locked by Java, preventing subsequent updates until a manual restart of the slave process.

      Our Java version is rather old, so I'm going to try updating that, in case there's a problem in the mechanism by which it transfers the file over the network.

            jglick Jesse Glick
            krandell Kim Randell
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: