Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-70378

Add support for "stream at change"

    XMLWordPrintable

Details

    Description

      Add support for "stream at change" in the p4sync step

      Use case is where we want to build code from a few months ago but the stream spec has now changed.
      So although we can specify the code to be build using a label/cl/timestamp we somewhow also need to indicate that we want to use the stream at a particular change

      Attachments

        Activity

          mornejoubert Morne Joubert added a comment -

          But do you agree that if we want to make 100% sure we can build older items we need

          1) The client spec as it was at point X
          2) The stream(s) at point X
          3) the code at point X

          AFAIK if you currently tell the plugin to us a label we only apply that label to point 3, correct ?

          mornejoubert Morne Joubert added a comment - But do you agree that if we want to make 100% sure we can build older items we need 1) The client spec as it was at point X 2) The stream(s) at point X 3) the code at point X AFAIK if you currently tell the plugin to us a label we only apply that label to point 3, correct ?
          p4karl Karl Wirth added a comment -

          Hi mornejoubert - Are you using Stream specs for the client workspace definition? With the P4 client, sync at change will do (1), (2) and (3).

          I want to sync stream path at changelist 1234, this pulls the stream spec out of the tables at the correct point in time which is used to created the client spec which then is used to sync the files at changelist 1234 (If you have 'dm.sync.streamchange=1' set ).

          At least that's what my testing is showing and how I read the docs.

          Regards,

          Karl

          p4karl Karl Wirth added a comment - Hi mornejoubert - Are you using Stream specs for the client workspace definition? With the P4 client, sync at change will do (1), (2) and (3). I want to sync stream path at changelist 1234, this pulls the stream spec out of the tables at the correct point in time which is used to created the client spec which then is used to sync the files at changelist 1234 (If you have ' dm.sync.streamchange=1' set ). At least that's what my testing is showing and how I read the docs. Regards, Karl
          p4karl Karl Wirth added a comment -

          Hi mornejoubert - I confirm this feature does not work in P4Jenkins with virtual streams

           

          Also I confirm that when I use a CL specifier with sync the sync uses the streams view at the CL:

          $ p4 sync ...@167
          Stream client view derived from stream spec version at or before change 167 for stream //streams/Virtual.
          //streams/main/f168#1 - deleted as /ws_stream/f168

          p4karl Karl Wirth added a comment - Hi mornejoubert - I confirm this feature does not work in P4Jenkins with virtual streams   Also I confirm that when I use a CL specifier with sync the sync uses the streams view at the CL: $ p4 sync ...@167 Stream client view derived from stream spec version at or before change 167 for stream //streams/Virtual. //streams/main/f168#1 - deleted as /ws_stream/f168
          mornejoubert Morne Joubert added a comment -

          Hopefully i am using the correct terminology

          I have a workspace (virtual stream spec) called X that i use to build our code.

          14:27:12  Owner                    :jenkins
          14:27:12  Access                   :2023/01/30 14:32:47
          14:27:12  Update                   :2023/03/10 15:27:12
          14:27:12  SubmitOptions            :submitunchanged
          14:27:12  View14                   ://MA_SW/tools/metrics/... //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/ma_tools/metrics/...
          14:27:12  View15                   ://MA_SW/config/... //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/__CONFIG__/...
          14:27:12  View16                   ://depot/jenkins/pipelines/... //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/jenkins/pipelines/...
          14:27:12  View17                   ://sdmos/ci_sdmos/%%1 //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/sdmos/%%1
          14:27:12  View18                   ://sdmos/ci_sdmos/tools/... //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/sdmos/tools/...
          14:27:12  View19                   ://sdmos/ci_sdmos/tools/performance/analyse_profiling_logs.py //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/atf/scripts/utils/analyse_profiling_logs.py
          14:27:12  extraTagType0            :words
          14:27:12  Stream                   ://atf/dev_atf_tm_pts
          14:27:12  Client                   :jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0
          14:27:12  View10                   ://tm/main_pixit/... //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/pts/pixit/...
          14:27:12  View11                   ://buildsystem/cmake/... //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/cbs/...
          14:27:12  View12                   ://atf/int_mlb/ma_tools/... //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/ma_tools/...
          14:27:12  View13                   ://MA_SW/tools/bin/... //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/ma_tools/bin/...
          14:27:12  Root                     :c:\jenkins\ws\AdevAtmptsitsgndt5830
          14:27:12  ServerID                 :sgn.2
          14:27:12  Options                  :noallwrite clobber nocompress unlocked modtime rmdir
          14:27:12  Description              :2023-01-30-13:13:36
          14:27:12  
          14:27:12  Backup                   :enable
          14:27:12  View2                    ://tm/dev_testlink/libs/Import_UBXLib.py //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/atf/scripts/testlink/Import_UBXLib.py
          14:27:12  extraTag0                :writable
          14:27:12  View1                    ://MA_SW/int_thread_km/km_system/builds/ps_app/keys/... //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/atf/km_system/builds/ps_app/keys/...
          14:27:12  View4                    ://tm/dev_testlink/libs/SQL_UBXLib.py //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/atf/scripts/testlink/SQL_UBXLib.py
          14:27:12  View3                    ://tm/dev_testlink/scripts/ParseTMTestReports.py //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/atf/scripts/testlink/ParseTMTestReports.py
          14:27:12  LineEnd                  :local
          14:27:12  View0                    ://atf/int_mlb/... //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/atf/...
          14:27:12  writable3                ://tm/ci_pts/...
          14:27:12  writable2                ://MA_SW/tools/bin/...
          14:27:12  writable1                ://MA_SW/tools/metrics/...
          14:27:12  writable0                ://tm/dev_testlink/scripts/ParseTMTestReports.py
          14:27:12  View25                   :-//atf/....pyc //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/....pyc
          14:27:12  View26                   :-//atf/.../nexus/tools/KM_TO2/bootloaders/1.1.2/... //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/.../nexus/tools/KM_TO2/bootloaders/1.1.2/...
          14:27:12  View27                   :-//atf/.../nexus_downloads/... //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/.../nexus_downloads/...
          14:27:12  Type                     :writeable
          14:27:12  View28                   :-//atf/.../scripts/testers/rscontest/test_... //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/.../scripts/testers/rscontest/test_...
          14:27:12  View29                   :-//atf/.../scripts/utils/test_files/... //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/.../scripts/utils/test_files/...
          14:27:12  View9                    ://tm/ci_pts/... //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/pts/...
          14:27:12  View20                   ://sdmos/ci_sdmos/tools/coredump/... //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/atf/scripts/utils/coredump/...
          14:27:12  View6                    ://tm/dev_testlink/libs/nexus.py //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/atf/scripts/testlink/nexus.py
          14:27:12  View21                   ://sdmos/ci_sdmos/tools/kmcoredump/... //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/atf/scripts/utils/kmcoredump/...
          14:27:12  View5                    ://tm/dev_testlink/libs/XLSX_UBXLib.py //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/atf/scripts/testlink/XLSX_UBXLib
          14:27:12  View22                   ://MA_SW/tools/metrics/jenkins_master_tokens.py //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/atf/scripts/utils/jenkins_master_tokens.py
          14:27:12  View8                    ://ltf/dev_uLogR_build/src/utilities/... //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/atf/scripts/ulogr/...
          14:27:12  View23                   ://atf/int_mlb/atf_env/... //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/atf/atf_env_old/...
          14:27:12  View7                    ://tm/dev_testlink/libs/testlinklib.py //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/atf/scripts/testlink/testlinklib.py
          14:27:12  View24                   ://ubx/tokeniser/... //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/tokeniser/...
          

          This virtual stream is a child of
          Stream ://atf/dev_atf_tm_pts
          and imports files from various other streams

          What i can deduce from the the code,logs is that the plugin will first make a new copy of X into a new workspace based on naming conventions e.g. X_1  and do the root/mappings changes relevant for the particular jenkins workspaces,node names etc.

          So at this point we have a new workspace X_1.

          This X_1 is then used for the sync, revert, reconcile etc.
           

          So my first question is that if X have been changed a few times would the p4 plugin known that if we sync to a timestamp of last year to use X of last year and not X of today (to create X_1) ?

          Would
          p4 -c X_1 sync --use-stream-change @2011/06/24

          Use the X of today or X@2011/06/24 to sync the code according to where all the streams was at this point ?

           

           

           

          mornejoubert Morne Joubert added a comment - Hopefully i am using the correct terminology I have a workspace (virtual stream spec) called X that i use to build our code. 14:27:12 Owner :jenkins 14:27:12 Access :2023/01/30 14:32:47 14:27:12 Update :2023/03/10 15:27:12 14:27:12 SubmitOptions :submitunchanged 14:27:12 View14 ://MA_SW/tools/metrics/... //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/ma_tools/metrics/... 14:27:12 View15 ://MA_SW/config/... //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/__CONFIG__/... 14:27:12 View16 ://depot/jenkins/pipelines/... //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/jenkins/pipelines/... 14:27:12 View17 ://sdmos/ci_sdmos/%%1 //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/sdmos/%%1 14:27:12 View18 ://sdmos/ci_sdmos/tools/... //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/sdmos/tools/... 14:27:12 View19 ://sdmos/ci_sdmos/tools/performance/analyse_profiling_logs.py //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/atf/scripts/utils/analyse_profiling_logs.py 14:27:12 extraTagType0 :words 14:27:12 Stream ://atf/dev_atf_tm_pts 14:27:12 Client :jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0 14:27:12 View10 ://tm/main_pixit/... //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/pts/pixit/... 14:27:12 View11 ://buildsystem/cmake/... //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/cbs/... 14:27:12 View12 ://atf/int_mlb/ma_tools/... //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/ma_tools/... 14:27:12 View13 ://MA_SW/tools/bin/... //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/ma_tools/bin/... 14:27:12 Root :c:\jenkins\ws\AdevAtmptsitsgndt5830 14:27:12 ServerID :sgn.2 14:27:12 Options :noallwrite clobber nocompress unlocked modtime rmdir 14:27:12 Description :2023-01-30-13:13:36 14:27:12 14:27:12 Backup :enable 14:27:12 View2 ://tm/dev_testlink/libs/Import_UBXLib.py //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/atf/scripts/testlink/Import_UBXLib.py 14:27:12 extraTag0 :writable 14:27:12 View1 ://MA_SW/int_thread_km/km_system/builds/ps_app/keys/... //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/atf/km_system/builds/ps_app/keys/... 14:27:12 View4 ://tm/dev_testlink/libs/SQL_UBXLib.py //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/atf/scripts/testlink/SQL_UBXLib.py 14:27:12 View3 ://tm/dev_testlink/scripts/ParseTMTestReports.py //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/atf/scripts/testlink/ParseTMTestReports.py 14:27:12 LineEnd :local 14:27:12 View0 ://atf/int_mlb/... //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/atf/... 14:27:12 writable3 ://tm/ci_pts/... 14:27:12 writable2 ://MA_SW/tools/bin/... 14:27:12 writable1 ://MA_SW/tools/metrics/... 14:27:12 writable0 ://tm/dev_testlink/scripts/ParseTMTestReports.py 14:27:12 View25 :-//atf/....pyc //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/....pyc 14:27:12 View26 :-//atf/.../nexus/tools/KM_TO2/bootloaders/1.1.2/... //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/.../nexus/tools/KM_TO2/bootloaders/1.1.2/... 14:27:12 View27 :-//atf/.../nexus_downloads/... //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/.../nexus_downloads/... 14:27:12 Type :writeable 14:27:12 View28 :-//atf/.../scripts/testers/rscontest/test_... //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/.../scripts/testers/rscontest/test_... 14:27:12 View29 :-//atf/.../scripts/utils/test_files/... //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/.../scripts/utils/test_files/... 14:27:12 View9 ://tm/ci_pts/... //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/pts/... 14:27:12 View20 ://sdmos/ci_sdmos/tools/coredump/... //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/atf/scripts/utils/coredump/... 14:27:12 View6 ://tm/dev_testlink/libs/nexus.py //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/atf/scripts/testlink/nexus.py 14:27:12 View21 ://sdmos/ci_sdmos/tools/kmcoredump/... //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/atf/scripts/utils/kmcoredump/... 14:27:12 View5 ://tm/dev_testlink/libs/XLSX_UBXLib.py //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/atf/scripts/testlink/XLSX_UBXLib 14:27:12 View22 ://MA_SW/tools/metrics/jenkins_master_tokens.py //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/atf/scripts/utils/jenkins_master_tokens.py 14:27:12 View8 ://ltf/dev_uLogR_build/src/utilities/... //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/atf/scripts/ulogr/... 14:27:12 View23 ://atf/int_mlb/atf_env/... //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/atf/atf_env_old/... 14:27:12 View7 ://tm/dev_testlink/libs/testlinklib.py //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/atf/scripts/testlink/testlinklib.py 14:27:12 View24 ://ubx/tokeniser/... //jenkins-sgn-it-sgn-dt-583-__atf_dev_atf_tm_pts0/tokeniser/... This virtual stream is a child of Stream ://atf/dev_atf_tm_pts and imports files from various other streams What i can deduce from the the code,logs is that the plugin will first make a new copy of X into a new workspace based on naming conventions e.g. X_1   and do the root/mappings changes relevant for the particular jenkins workspaces,node names etc. So at this point we have a new workspace X_1 . This X_1 is then used for the sync, revert, reconcile etc.   So my first question is that if X have been changed a few times would the p4 plugin known that if we sync to a timestamp of last year to use X of last year and not X of today (to create X_1) ? Would p4 -c X_1 sync --use-stream-change @2011/06/24 Use the X of today or X@2011/06/24 to sync the code according to where all the streams was at this point ?      
          mornejoubert Morne Joubert added a comment -

          if i wanted to provide the feature via the p4 groovy steps, how would i do it ?

          mornejoubert Morne Joubert added a comment - if i wanted to provide the feature via the p4 groovy steps, how would i do it ?

          People

            Unassigned Unassigned
            mornejoubert Morne Joubert
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: