• Icon: Improvement Improvement
    • Resolution: Unresolved
    • Icon: Minor Minor
    • git-plugin
    • None
    • Fedora 33, Btrfs, Jenkins 2.277.2 all plugins updated

      I have a declarative pipeline with a docker agent running on a machine with SELinux enabled. At some point in the "checkout scm" step "git checkout -f <commit_id>" is run.

      During the first run the checkout is all fine. The Docker agent gives the correct SELinux label to the files and it's all fine. The problem is that when setting the SELinux label the files ctime is modified (even if the correct SELinux label was already in place) and in the next job run, if it's an incremental build "git checkout -f <commit_id>" updates the mtime... so the build system ends up rebuilding stuff that doesn't need to be rebuilt, making the build not that "incremental".

       

      Running "git update-index --refresh" the git index is updated and later "git checkout -f <commit_id>" doesn't update the mtime any more. But I don't really know git so well, so I don't know if running "git update-index --refresh" is a (good) fix.

       

          [JENKINS-65395] Docker/SELinux/git break incremental builds

          Mark Waite added a comment -

          jimklimov has been extending the git client plugin to better support SELinux. He notes that his changes are not yet fully functional in all cases, but are progressing. See the git client plugin 3.7.0 changelog for notes and PR-693 for a proposed addition.

          Mark Waite added a comment - jimklimov has been extending the git client plugin to better support SELinux. He notes that his changes are not yet fully functional in all cases, but are progressing. See the git client plugin 3.7.0 changelog for notes and PR-693 for a proposed addition.

          Mark Waite added a comment -

          I generally expect that command line git will manage file time stamps. I hesitate to add additional calls to command line git in order to avoid file timestamp issues. If you need an operation like git update-index, I recommend you perform that operation after checkout is complete and before the start of other steps.

          Mark Waite added a comment - I generally expect that command line git will manage file time stamps. I hesitate to add additional calls to command line git in order to avoid file timestamp issues. If you need an operation like git update-index , I recommend you perform that operation after checkout is complete and before the start of other steps.

          Jim Klimov added a comment -

          Looking at the description, I do not think the extensions I did so far are related: they set the label on temporary file (in a temporary directory like `.../workspace/jobname@tmp/`) used to pass the SSH key, which is removed after the git operation. Beside what Mark suggested, I would try (if possible) to run a build with SELinux disabled, or on a non-SELinux system at all, to see if the builds "mis-"behave the same? I suppose for timestamps in the git workspace, if the timestamping happens due to git itself, the behavior would not change.

          I do not have access to a SELinux installation today, but would recommend double-checking on yours with the `stat` command whether `chcon` really changes the timestamps, and if it does - perhaps bring it up as a bug with the #selinux channel on Freenode IRC or on other media.

          And also look at Docker in the equation. While not a user, I heard it can be set up with persistent storage, but by default is discardable. So double-check that what you assume to be incremental builds are not in fact always first builds done from scratch?

          Jim Klimov added a comment - Looking at the description, I do not think the extensions I did so far are related: they set the label on temporary file (in a temporary directory like `.../workspace/jobname@tmp/`) used to pass the SSH key, which is removed after the git operation. Beside what Mark suggested, I would try (if possible) to run a build with SELinux disabled, or on a non-SELinux system at all, to see if the builds "mis-"behave the same? I suppose for timestamps in the git workspace, if the timestamping happens due to git itself, the behavior would not change. I do not have access to a SELinux installation today, but would recommend double-checking on yours with the `stat` command whether `chcon` really changes the timestamps, and if it does - perhaps bring it up as a bug with the #selinux channel on Freenode IRC or on other media. And also look at Docker in the equation. While not a user, I heard it can be set up with persistent storage, but by default is discardable. So double-check that what you assume to be incremental builds are not in fact always first builds done from scratch?

          Cristian added a comment -

          I would try (if possible) to run a build with SELinux disabled, or on a non-SELinux system at all, to see if the builds "mis-"behave the same? I suppose for timestamps in the git workspace, if the timestamping happens due to git itself, the behavior would not change.

          With SELinux disabled the incremental builds works fine (the logs shows nothing is rebuilt, the job completes way faster). "stat" confirms the files mtime/ctime don't change.

           

          I do not have access to a SELinux installation today, but would recommend double-checking on yours with the `stat` command whether `chcon` really changes the timestamps, and if it does - perhaps bring it up as a bug with the #selinux channel on Freenode IRC or on other media.

          I did, it changes it (ctime only, not mtime... mtime is changed by the "git checkout" later). As far as I know it's the correct behaviour. If you search for "SELinux ctime" you actually can find people complaining about the ctime not being modified (e.g. https://bugzilla.redhat.com/show_bug.cgi?id=1051736).

           

          And also look at Docker in the equation. While not a user, I heard it can be set up with persistent storage, but by default is discardable. So double-check that what you assume to be incremental builds are not in fact always first builds done from scratch?

          The Docker plugin creates a bind mount of the Jenkins workspace into the container, so it's "persistent storage". It is indeed an incremental build.

           

          Cristian added a comment - I would try (if possible) to run a build with SELinux disabled, or on a non-SELinux system at all, to see if the builds "mis-"behave the same? I suppose for timestamps in the git workspace, if the timestamping happens due to git itself, the behavior would not change. With SELinux disabled the incremental builds works fine (the logs shows nothing is rebuilt, the job completes way faster). "stat" confirms the files mtime/ctime don't change.   I do not have access to a SELinux installation today, but would recommend double-checking on yours with the `stat` command whether `chcon` really changes the timestamps, and if it does - perhaps bring it up as a bug with the #selinux channel on Freenode IRC or on other media. I did, it changes it (ctime only, not mtime... mtime is changed by the "git checkout" later). As far as I know it's the correct behaviour. If you search for "SELinux ctime" you actually can find people complaining about the ctime not being modified (e.g. https://bugzilla.redhat.com/show_bug.cgi?id=1051736 ).   And also look at Docker in the equation. While not a user, I heard it can be set up with persistent storage, but by default is discardable. So double-check that what you assume to be incremental builds are not in fact always first builds done from scratch? The Docker plugin creates a bind mount of the Jenkins workspace into the container, so it's "persistent storage". It is indeed an incremental build.  

          Cristian added a comment - - edited

          I have created a dummy repository with a single commit with a single file.

          A "git checkout" does nothing.

          [reddwarf@localhost jenkins_git_test]$ git rev-parse HEAD
          e7929c5cb1dca67a733d8cc5267ab7cd4cfa5c02
          [reddwarf@localhost jenkins_git_test]$ git diff-index HEAD
          [reddwarf@localhost jenkins_git_test]$ stat dummy
            File: dummy
            Size: 0         	Blocks: 0          IO Block: 4096   regular empty file
          Device: 34h/52d	Inode: 39266865    Links: 1
          Access: (0664/-rw-rw-r--)  Uid: ( 1000/reddwarf)   Gid: ( 1000/reddwarf)
          Context: unconfined_u:object_r:user_home_t:s0
          Access: 2021-04-18 21:54:36.058747385 +0100
          Modify: 2021-04-18 21:54:36.058747385 +0100
          Change: 2021-04-18 21:54:36.058747385 +0100
           Birth: 2021-04-18 21:54:18.314658902 +0100
          [reddwarf@localhost jenkins_git_test]$ git checkout -f e7929c5cb1dca67a733d8cc5267ab7cd4cfa5c02
          Note: switching to 'e7929c5cb1dca67a733d8cc5267ab7cd4cfa5c02'.You are in 'detached HEAD' state. You can look around, make experimental
          changes and commit them, and you can discard any commits you make in this
          state without impacting any branches by switching back to a branch.If you want to create a new branch to retain commits you create, you may
          do so (now or later) by using -c with the switch command. Example:  git switch -c <new-branch-name>Or undo this operation with:  git switch -Turn off this advice by setting config variable advice.detachedHead to falseHEAD is now at e7929c5 Dummy
          [reddwarf@localhost jenkins_git_test]$ git diff-index HEAD
          [reddwarf@localhost jenkins_git_test]$ stat dummy
            File: dummy
            Size: 0         	Blocks: 0          IO Block: 4096   regular empty file
          Device: 34h/52d	Inode: 39266865    Links: 1
          Access: (0664/-rw-rw-r--)  Uid: ( 1000/reddwarf)   Gid: ( 1000/reddwarf)
          Context: unconfined_u:object_r:user_home_t:s0
          Access: 2021-04-18 21:54:36.058747385 +0100
          Modify: 2021-04-18 21:54:36.058747385 +0100
          Change: 2021-04-18 21:54:36.058747385 +0100
           Birth: 2021-04-18 21:54:18.314658902 +0100
          [reddwarf@localhost jenkins_git_test]$ 

          chcon changes the ctime, the following git checkout changes the mtime

          [reddwarf@localhost jenkins_git_test]$ chcon system_u:object_r:container_file_t:s0 dummy
          [reddwarf@localhost jenkins_git_test]$ stat dummy
            File: dummy
            Size: 0         	Blocks: 0          IO Block: 4096   regular empty file
          Device: 34h/52d	Inode: 39266865    Links: 1
          Access: (0664/-rw-rw-r--)  Uid: ( 1000/reddwarf)   Gid: ( 1000/reddwarf)
          Context: system_u:object_r:container_file_t:s0
          Access: 2021-04-18 21:54:36.058747385 +0100
          Modify: 2021-04-18 21:54:36.058747385 +0100
          Change: 2021-04-18 21:56:00.269096768 +0100
           Birth: 2021-04-18 21:54:18.314658902 +0100
          [reddwarf@localhost jenkins_git_test]$ git diff-index HEAD
          :100644 100644 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 0000000000000000000000000000000000000000 M	dummy
          [reddwarf@localhost jenkins_git_test]$ git checkout -f e7929c5cb1dca67a733d8cc5267ab7cd4cfa5c02
          HEAD is now at e7929c5 Dummy
          [reddwarf@localhost jenkins_git_test]$ stat dummy
            File: dummy
            Size: 0         	Blocks: 0          IO Block: 4096   regular empty file
          Device: 34h/52d	Inode: 39266933    Links: 1
          Access: (0664/-rw-rw-r--)  Uid: ( 1000/reddwarf)   Gid: ( 1000/reddwarf)
          Context: unconfined_u:object_r:user_home_t:s0
          Access: 2021-04-18 21:56:16.493161885 +0100
          Modify: 2021-04-18 21:56:16.493161885 +0100
          Change: 2021-04-18 21:56:16.493161885 +0100
           Birth: 2021-04-18 21:56:16.493161885 +0100
          [reddwarf@localhost jenkins_git_test]$ git diff-index HEAD
          [reddwarf@localhost jenkins_git_test]$ 

          The same thing with a "git update-index --refresh" in the middle results in the mtime not changing

          [reddwarf@localhost jenkins_git_test]$ chcon system_u:object_r:container_file_t:s0 dummy
          [reddwarf@localhost jenkins_git_test]$ stat dummy
            File: dummy
            Size: 0         	Blocks: 0          IO Block: 4096   regular empty file
          Device: 34h/52d	Inode: 39266933    Links: 1
          Access: (0664/-rw-rw-r--)  Uid: ( 1000/reddwarf)   Gid: ( 1000/reddwarf)
          Context: system_u:object_r:container_file_t:s0
          Access: 2021-04-18 21:56:16.493161885 +0100
          Modify: 2021-04-18 21:56:16.493161885 +0100
          Change: 2021-04-18 21:56:33.875231655 +0100
           Birth: 2021-04-18 21:56:16.493161885 +0100
          [reddwarf@localhost jenkins_git_test]$ git diff-index HEAD
          :100644 100644 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 0000000000000000000000000000000000000000 M	dummy
          [reddwarf@localhost jenkins_git_test]$ git update-index --refresh
          [reddwarf@localhost jenkins_git_test]$ stat dummy
            File: dummy
            Size: 0         	Blocks: 0          IO Block: 4096   regular empty file
          Device: 34h/52d	Inode: 39266933    Links: 1
          Access: (0664/-rw-rw-r--)  Uid: ( 1000/reddwarf)   Gid: ( 1000/reddwarf)
          Context: system_u:object_r:container_file_t:s0
          Access: 2021-04-18 21:56:16.493161885 +0100
          Modify: 2021-04-18 21:56:16.493161885 +0100
          Change: 2021-04-18 21:56:33.875231655 +0100
           Birth: 2021-04-18 21:56:16.493161885 +0100
          [reddwarf@localhost jenkins_git_test]$ git checkout -f e7929c5cb1dca67a733d8cc5267ab7cd4cfa5c02
          HEAD is now at e7929c5 Dummy
          [reddwarf@localhost jenkins_git_test]$ stat dummy
            File: dummy
            Size: 0         	Blocks: 0          IO Block: 4096   regular empty file
          Device: 34h/52d	Inode: 39266933    Links: 1
          Access: (0664/-rw-rw-r--)  Uid: ( 1000/reddwarf)   Gid: ( 1000/reddwarf)
          Context: system_u:object_r:container_file_t:s0
          Access: 2021-04-18 21:56:16.493161885 +0100
          Modify: 2021-04-18 21:56:16.493161885 +0100
          Change: 2021-04-18 21:56:33.875231655 +0100
           Birth: 2021-04-18 21:56:16.493161885 +0100
          

          Cristian added a comment - - edited I have created a dummy repository with a single commit with a single file. A "git checkout" does nothing. [reddwarf@localhost jenkins_git_test]$ git rev-parse HEAD e7929c5cb1dca67a733d8cc5267ab7cd4cfa5c02 [reddwarf@localhost jenkins_git_test]$ git diff-index HEAD [reddwarf@localhost jenkins_git_test]$ stat dummy File: dummy Size: 0 Blocks: 0 IO Block: 4096 regular empty file Device: 34h/52d Inode: 39266865 Links: 1 Access: (0664/-rw-rw-r--) Uid: ( 1000/reddwarf) Gid: ( 1000/reddwarf) Context: unconfined_u:object_r:user_home_t:s0 Access: 2021-04-18 21:54:36.058747385 +0100 Modify: 2021-04-18 21:54:36.058747385 +0100 Change: 2021-04-18 21:54:36.058747385 +0100 Birth: 2021-04-18 21:54:18.314658902 +0100 [reddwarf@localhost jenkins_git_test]$ git checkout -f e7929c5cb1dca67a733d8cc5267ab7cd4cfa5c02 Note: switching to 'e7929c5cb1dca67a733d8cc5267ab7cd4cfa5c02'.You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by switching back to a branch.If you want to create a new branch to retain commits you create, you may do so (now or later) by using -c with the switch command. Example: git switch -c <new-branch-name>Or undo this operation with: git switch -Turn off this advice by setting config variable advice.detachedHead to falseHEAD is now at e7929c5 Dummy [reddwarf@localhost jenkins_git_test]$ git diff-index HEAD [reddwarf@localhost jenkins_git_test]$ stat dummy File: dummy Size: 0 Blocks: 0 IO Block: 4096 regular empty file Device: 34h/52d Inode: 39266865 Links: 1 Access: (0664/-rw-rw-r--) Uid: ( 1000/reddwarf) Gid: ( 1000/reddwarf) Context: unconfined_u:object_r:user_home_t:s0 Access: 2021-04-18 21:54:36.058747385 +0100 Modify: 2021-04-18 21:54:36.058747385 +0100 Change: 2021-04-18 21:54:36.058747385 +0100 Birth: 2021-04-18 21:54:18.314658902 +0100 [reddwarf@localhost jenkins_git_test]$ chcon changes the ctime, the following git checkout changes the mtime [reddwarf@localhost jenkins_git_test]$ chcon system_u:object_r:container_file_t:s0 dummy [reddwarf@localhost jenkins_git_test]$ stat dummy File: dummy Size: 0 Blocks: 0 IO Block: 4096 regular empty file Device: 34h/52d Inode: 39266865 Links: 1 Access: (0664/-rw-rw-r--) Uid: ( 1000/reddwarf) Gid: ( 1000/reddwarf) Context: system_u:object_r:container_file_t:s0 Access: 2021-04-18 21:54:36.058747385 +0100 Modify: 2021-04-18 21:54:36.058747385 +0100 Change: 2021-04-18 21:56:00.269096768 +0100 Birth: 2021-04-18 21:54:18.314658902 +0100 [reddwarf@localhost jenkins_git_test]$ git diff-index HEAD :100644 100644 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 0000000000000000000000000000000000000000 M dummy [reddwarf@localhost jenkins_git_test]$ git checkout -f e7929c5cb1dca67a733d8cc5267ab7cd4cfa5c02 HEAD is now at e7929c5 Dummy [reddwarf@localhost jenkins_git_test]$ stat dummy File: dummy Size: 0 Blocks: 0 IO Block: 4096 regular empty file Device: 34h/52d Inode: 39266933 Links: 1 Access: (0664/-rw-rw-r--) Uid: ( 1000/reddwarf) Gid: ( 1000/reddwarf) Context: unconfined_u:object_r:user_home_t:s0 Access: 2021-04-18 21:56:16.493161885 +0100 Modify: 2021-04-18 21:56:16.493161885 +0100 Change: 2021-04-18 21:56:16.493161885 +0100 Birth: 2021-04-18 21:56:16.493161885 +0100 [reddwarf@localhost jenkins_git_test]$ git diff-index HEAD [reddwarf@localhost jenkins_git_test]$ The same thing with a "git update-index --refresh" in the middle results in the mtime not changing [reddwarf@localhost jenkins_git_test]$ chcon system_u:object_r:container_file_t:s0 dummy [reddwarf@localhost jenkins_git_test]$ stat dummy File: dummy Size: 0 Blocks: 0 IO Block: 4096 regular empty file Device: 34h/52d Inode: 39266933 Links: 1 Access: (0664/-rw-rw-r--) Uid: ( 1000/reddwarf) Gid: ( 1000/reddwarf) Context: system_u:object_r:container_file_t:s0 Access: 2021-04-18 21:56:16.493161885 +0100 Modify: 2021-04-18 21:56:16.493161885 +0100 Change: 2021-04-18 21:56:33.875231655 +0100 Birth: 2021-04-18 21:56:16.493161885 +0100 [reddwarf@localhost jenkins_git_test]$ git diff-index HEAD :100644 100644 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 0000000000000000000000000000000000000000 M dummy [reddwarf@localhost jenkins_git_test]$ git update-index --refresh [reddwarf@localhost jenkins_git_test]$ stat dummy File: dummy Size: 0 Blocks: 0 IO Block: 4096 regular empty file Device: 34h/52d Inode: 39266933 Links: 1 Access: (0664/-rw-rw-r--) Uid: ( 1000/reddwarf) Gid: ( 1000/reddwarf) Context: system_u:object_r:container_file_t:s0 Access: 2021-04-18 21:56:16.493161885 +0100 Modify: 2021-04-18 21:56:16.493161885 +0100 Change: 2021-04-18 21:56:33.875231655 +0100 Birth: 2021-04-18 21:56:16.493161885 +0100 [reddwarf@localhost jenkins_git_test]$ git checkout -f e7929c5cb1dca67a733d8cc5267ab7cd4cfa5c02 HEAD is now at e7929c5 Dummy [reddwarf@localhost jenkins_git_test]$ stat dummy File: dummy Size: 0 Blocks: 0 IO Block: 4096 regular empty file Device: 34h/52d Inode: 39266933 Links: 1 Access: (0664/-rw-rw-r--) Uid: ( 1000/reddwarf) Gid: ( 1000/reddwarf) Context: system_u:object_r:container_file_t:s0 Access: 2021-04-18 21:56:16.493161885 +0100 Modify: 2021-04-18 21:56:16.493161885 +0100 Change: 2021-04-18 21:56:33.875231655 +0100 Birth: 2021-04-18 21:56:16.493161885 +0100

          Cristian added a comment -

          And yes, using

          stage('Checkout') {
              steps {
                  sh label:  'Update git index',
                     script: '[ -d .git ] && git update-index --refresh || true'
                  checkout scm
              }
          }

          "Fixes" the issue for me.

          Cristian added a comment - And yes, using stage('Checkout') { steps { sh label: 'Update git index', script: '[ -d .git ] && git update-index --refresh || true' checkout scm } } "Fixes" the issue for me.

          Carl Verbiest added a comment -

          Same/similar issue, without docker, on Jenkins 2.414.2 with all plugins up-to-date. agent running on Red Hat Enterprise Linux release 8.5 with selinux enabled

          all files get their timestamp updated even though no commits were done to the repo.

          using scmGit in a jenkins shared library call.

           
          def call(Map config = [:]) {
              if (config.branch == '' || config.branch[0] == ':') {
                  print "No checkout for ${config.dir} config.branch=${config.branch}"
                  return
              }

              dir("${config.dir}") {
                  print "checkout scmGit ${config.dir} config.branch=${config.branch} url=${config.url} credentials=${config.credentialsId}"

                  checkout scmGit(
                      branches: [[name: "${config.branch}"]],
                      extensions: [ lfs(), [$class: 'LocalBranch', localBranch: "${config.branch}"] ],
                      userRemoteConfigs: [[ credentialsId: "${config.credentialsId}", url: "${config.url}"]])
              }
          }

          Carl Verbiest added a comment - Same/similar issue, without docker, on Jenkins 2.414.2 with all plugins up-to-date. agent running on Red Hat Enterprise Linux release 8.5 with selinux enabled all files get their timestamp updated even though no commits were done to the repo. using scmGit in a jenkins shared library call.   def call(Map config = [:] ) {     if (config.branch == '' || config.branch [0] == ':') {         print "No checkout for ${config.dir} config.branch=${config.branch}"         return     }     dir("${config.dir}") {         print "checkout scmGit ${config.dir} config.branch=${config.branch} url=${config.url} credentials=${config.credentialsId}"         checkout scmGit(             branches: [ [name: "${config.branch}"] ],             extensions: [ lfs(), [$class: 'LocalBranch', localBranch: "${config.branch}"] ],             userRemoteConfigs: [[ credentialsId: "${config.credentialsId}", url: "${config.url}"]])     } }

            Unassigned Unassigned
            reddwarf94 Cristian
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: