• Icon: Improvement Improvement
    • Resolution: Fixed
    • Icon: Critical Critical
    • other
    • None
    • Platform: All, OS: All

      Seems that the usual way how jobs work on hudson is that there is one job for
      building a project and then if I want to have a subsequent job to run all unit
      tests, that job needs to download artifacts produced by that build job from
      Hudson. As a consequence, people usually add separate targets to their build
      scripts for running unit tests from hudson which download the build artifacts
      from the network.
      I think it would be nice if things could work the way how users would do it when
      running tests locally - i.e. use usual targets for building and then simply run
      a target for unit tests in the same workspace. This could be achieved by
      providing the subsequent jobs an exact snapshot of the workspace at the time
      when the parent job finished running.

          [JENKINS-682] Clone workspace between jobs

          mmatula created issue -

          mmatula added a comment -

          Oops, this an enhancement request, not a defect...

          mmatula added a comment - Oops, this an enhancement request, not a defect...

          The problem with doing this is that then you won't be able to run the main build
          and tests in parallel, so it increases the turn-around time.

          What's the difference between setting up two jobs with the same workspace, as
          opposed to setting up one job that does both the build and the test?

          Kohsuke Kawaguchi added a comment - The problem with doing this is that then you won't be able to run the main build and tests in parallel, so it increases the turn-around time. What's the difference between setting up two jobs with the same workspace, as opposed to setting up one job that does both the build and the test?

          mmatula added a comment -

          I see. I was not sure how it is implemented - i.e. whether different slaves
          access the same physical workspace. I thought each slave checks out its own
          workspace for a given job in which case tests and a new build could still run in
          parallel.
          I thought the difference between running it as a single job would be exactly
          that possibility to start another build before tests (and possibly other
          subsequent jobs such as coverage, findbugs, javadoc generation, etc.) finished
          running.

          mmatula added a comment - I see. I was not sure how it is implemented - i.e. whether different slaves access the same physical workspace. I thought each slave checks out its own workspace for a given job in which case tests and a new build could still run in parallel. I thought the difference between running it as a single job would be exactly that possibility to start another build before tests (and possibly other subsequent jobs such as coverage, findbugs, javadoc generation, etc.) finished running.

          Every slave uses a workspace on local file system, so different slaves get
          different workspaces.

          The problem I was trying to point out in sharing a workspace is that if a new
          build starts while a test run is in progress, it will most likely mess up. For
          example, a build might try to overwrite the file that a test is using. So in
          general, I don't see how you can reliably run multiple tasks over the same
          workspace in parallel.

          Or maybe what you are really saying is a slightly different model, where the
          execution will go like:

          ws #0 <--- build #N --><-- test #N --->
          ws #1 <--- build #N+1 --><--test #N+1 --->

          ?

          Kohsuke Kawaguchi added a comment - Every slave uses a workspace on local file system, so different slaves get different workspaces. The problem I was trying to point out in sharing a workspace is that if a new build starts while a test run is in progress, it will most likely mess up. For example, a build might try to overwrite the file that a test is using. So in general, I don't see how you can reliably run multiple tasks over the same workspace in parallel. Or maybe what you are really saying is a slightly different model, where the execution will go like: ws #0 <--- build #N -- >< -- test #N ---> ws #1 <--- build #N+1 -- >< --test #N+1 ---> ?

          gradopado added a comment -

          Have a look at Buildbot, where each build is composed of several build steps.
          Steps are tied to one another, no overlapping.

          The first one is mostly cvs-checkout and then you are free to do in steps what
          you think is necessary. Each step has its own result and derective how to
          proceed on WARNING and ERROR cases. This concept is nice.

          gradopado added a comment - Have a look at Buildbot, where each build is composed of several build steps. Steps are tied to one another, no overlapping. The first one is mostly cvs-checkout and then you are free to do in steps what you think is necessary. Each step has its own result and derective how to proceed on WARNING and ERROR cases. This concept is nice.

          mmatula added a comment -

          Kohsuke, what I would like is to be able to configure my job in such a way that
          it could assume the workspace is in the same state as it was after it's parent
          job. So, if some job is triggered by other job, then I thought it would be good
          if there was a way for this triggered job to inherit the workspace of that
          triggering job. So that people would not have to do additional "magic" specific
          to hudson (and maintain additional hudson-specific targets) in their build
          scripts. Hudson does know that a given job is triggered by another job and most
          likely it is going to use the artifacts that other job produces. So I thought it
          would be nice if those could be provided to these subsequent jobs automatically.

          As a user that does not know much about hudson internals, I cannot suggest how
          to implement it. I thought one way (the basic one) it could be done is what you
          showed in your ASCII diagram. Maybe other possible way could be to have a
          mechanism to push a snapshot of the workspace from a job to other jobs it
          triggers (potentially to other machines if those jobs run on other systems). I
          guess this would be least limiting when it comes to parallel execution.

          mmatula added a comment - Kohsuke, what I would like is to be able to configure my job in such a way that it could assume the workspace is in the same state as it was after it's parent job. So, if some job is triggered by other job, then I thought it would be good if there was a way for this triggered job to inherit the workspace of that triggering job. So that people would not have to do additional "magic" specific to hudson (and maintain additional hudson-specific targets) in their build scripts. Hudson does know that a given job is triggered by another job and most likely it is going to use the artifacts that other job produces. So I thought it would be nice if those could be provided to these subsequent jobs automatically. As a user that does not know much about hudson internals, I cannot suggest how to implement it. I thought one way (the basic one) it could be done is what you showed in your ASCII diagram. Maybe other possible way could be to have a mechanism to push a snapshot of the workspace from a job to other jobs it triggers (potentially to other machines if those jobs run on other systems). I guess this would be least limiting when it comes to parallel execution.

          A note to myself — recent introduction of Resource interface can be used as a
          lock mechanism for multiple jobs to use the same workspace.

          This is convenient for some use cases, like occasionally running lengthy-task
          (like "mvn site") where normally a quick CI build runs.

          Kohsuke Kawaguchi added a comment - A note to myself — recent introduction of Resource interface can be used as a lock mechanism for multiple jobs to use the same workspace. This is convenient for some use cases, like occasionally running lengthy-task (like "mvn site") where normally a quick CI build runs.

          akostadinov added a comment -

          I want to mention the case where workspace is on NFS.

          By many of the jobs we lose most of the time checking out the sources so being
          able to share workspace saves considerable amount time.

          That is in the case of a multiple configuration project or the job can be
          separated into few tasks.

          I think that by being able to mark arbitrary jobs as sequential that will work
          for us.

          akostadinov added a comment - I want to mention the case where workspace is on NFS. By many of the jobs we lose most of the time checking out the sources so being able to share workspace saves considerable amount time. That is in the case of a multiple configuration project or the job can be separated into few tasks. I think that by being able to mark arbitrary jobs as sequential that will work for us.

          akostadinov added a comment -

          adding myself to the cc

          akostadinov added a comment - adding myself to the cc

            abayer Andrew Bayer
            mmatula mmatula
            Votes:
            42 Vote for this issue
            Watchers:
            30 Start watching this issue

              Created:
              Updated:
              Resolved: