Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-38722

Build is triggered after workspace deletion even without scm changes

    XMLWordPrintable

Details

    Description

      See second comment for updated issue cause.

      I have configured project, where in csm I have Additional Behaviours -> Pooling ignores commits in certain paths with Included Regions pointing to one specific file. I have there also additional behaviour "Clean before checkout".

      When post-receive hook triggers change, everything works as expected.

      Pool SCM schedule is set to "H/1 * * * *" (each hour).

      Yesterday I have found that build has been started in night despite no changes was made to given file. In pool log for given build is this:
      NOTE: In second comment https://issues.jenkins-ci.org/browse/JENKINS-38722?focusedCommentId=272666&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-272666 is actual pooling log of problematic build triggering. This build has been started by something else (I do not know by what), but it is very similar to two referenced similar issues JENKINS-20389 and JENKINS-23606.

      Started on Oct 4, 2016 3:11:54 PM
      Polling SCM changes on master
      Using strategy: Default
      [poll] Last Built Revision: Revision 5eb3f4c485189f8151d0fe777a31bcfc48adb09e (origin/master)
       > git rev-parse --is-inside-work-tree # timeout=10
      Fetching changes from the remote Git repositories
       > git config remote.origin.url ssh://git@dev.asdf.sk/~/salt.git/ # timeout=10
      Cleaning workspace
       > git rev-parse --verify HEAD # timeout=10
      Resetting working tree
       > git reset --hard # timeout=10
       > git clean -fdx # timeout=10
      Fetching upstream changes from ssh://git@dev.asdf.sk/~/salt.git/
       > git --version # timeout=10
       > git fetch --tags --progress ssh://git@dev.asdf.sk/~/salt.git/ +refs/heads/*:refs/remotes/origin/*
      Polling for changes in
       > git rev-parse origin/master^{commit} # timeout=10
       > git log --full-history --no-abbrev --format=raw -M -m --raw 5eb3f4c485189f8151d0fe777a31bcfc48adb09e..abf529a33a519c6c712a419b19ac969d20896d6f # timeout=10
      Done. Took 0.46 sec
      Changes found
      

      In console output is this:

      Started by an SCM change
      [EnvInject] - Loading node environment variables.
      Building on master in workspace /var/lib/jenkins/workspace/autodeploy-stagebackend
       > git rev-parse --is-inside-work-tree # timeout=10
      Fetching changes from the remote Git repository
       > git config remote.origin.url ssh://git@dev.asdf.sk/~/salt.git/ # timeout=10
      Cleaning workspace
       > git rev-parse --verify HEAD # timeout=10
      Resetting working tree
       > git reset --hard # timeout=10
       > git clean -fdx # timeout=10
      Fetching upstream changes from ssh://git@dev.asdf.sk/~/salt.git/
       > git --version # timeout=10
       > git fetch --tags --progress ssh://git@dev.asdf.sk/~/salt.git/ +refs/heads/*:refs/remotes/origin/*
       > git rev-parse origin/master^{commit} # timeout=10
      Checking out Revision abf529a33a519c6c712a419b19ac969d20896d6f (origin/master)
       > git config core.sparsecheckout # timeout=10
       > git checkout -f abf529a33a519c6c712a419b19ac969d20896d6f
       > git rev-list 5eb3f4c485189f8151d0fe777a31bcfc48adb09e # timeout=10
      No emails were triggered.
      [autodeploy-stagebackend] $ /bin/bash -xe /tmp/hudson3936035578572855045.sh
      ....
      

      I think that pool log may sometime indicates ("No workspace is available, so can’t check for updates.") use of git ls-remote (https://wiki.jenkins-ci.org/display/JENKINS/Git+Plugin#GitPlugin-FastRemotePolling), So from morning I have added additional behaviour "force pooling using workspace". I hope it will help a bit tracking issue down.

      What has been done in git:

      5eb3f4c485189f8151d0fe777a31bcfc48adb09e last successfully build commit. It included file mentioned in included regions. It is commit with message "PRH-1542 New version of some app for stage"
      abf529a33a519c6c712a419b19ac969d2089 merge commit which merges given commit (rebase was not used when git pull was done, thus merge was done)

      Output of git log --full-history --no-abbrev --format=raw -M -m --raw 5eb3f4c485189f8151d0fe777a31bcfc48adb09e..abf529a33a519c6c712a419b19ac969d20896d6f

      commit abf529a33a519c6c712a419b19ac969d20896d6f (from 0a495d7f6b5a8db46edd2ef72a1bf9753ed87a84)
      tree 7f00c97840e8d6243362976310301c1c9dc70c7d
      parent 0a495d7f6b5a8db46edd2ef72a1bf9753ed87a84
      parent 5eb3f4c485189f8151d0fe777a31bcfc48adb09e
      author L X <lx@asdf.sk> 1475586630 +0200
      committer L X <lx@asdf.sk> 1475586630 +0200
      
          Merge branch 'master' of ssh://dev.asdf.sk/~/salt
          
          * 'master' of ssh://dev.asdf.sk/~/salt:
            PRH-1542 New version of bonus-data for stage.
      
      :100644 100644 3861e9a150a7f4d4713090575aeb04b85ec304cc c3fba55d424f5ea53103a088c30860438adf27ec M      pillar/backend/stage/init.sls
      
      commit abf529a33a519c6c712a419b19ac969d20896d6f (from 5eb3f4c485189f8151d0fe777a31bcfc48adb09e)
      tree 7f00c97840e8d6243362976310301c1c9dc70c7d
      parent 0a495d7f6b5a8db46edd2ef72a1bf9753ed87a84
      parent 5eb3f4c485189f8151d0fe777a31bcfc48adb09e
      author L X <lx@asdf.sk> 1475586630 +0200
      committer L X <lx@asdf.sk> 1475586630 +0200
      
          Merge branch 'master' of ssh://dev.asdf.sk/~/salt
          
          * 'master' of ssh://dev.asdf.sk/~/salt:
            PRH-1542 New version of some app for stage.
      
      :100644 100644 9ec0e5491222d6011af2859ccd2d476d02b84e2d bd9f2f6b08834e5196b6913b7cc1b07db8f94a4b M      pillar/centrala/production/asdf.sls
      
      commit 0a495d7f6b5a8db46edd2ef72a1bf9753ed87a84
      tree 9ce7c6cb816c5762e859a7932049d8b00feb5b9f
      parent 936b93fcb21501997450561a3c08b404dc610344
      author L X <lx@asdf.sk> 1475586451 +0200
      committer L X <lx@asdf.sk> 1475586451 +0200
      
          db_bonusy connect
      
      :100644 100644 9ec0e5491222d6011af2859ccd2d476d02b84e2d bd9f2f6b08834e5196b6913b7cc1b07db8f94a4b M      pillar/centrala/production/asdf.sls                          
      

      So I think that merge (wchich have not changed file since last build) trigered build. My suggestion would be to use git diff instead of git log for changes detection:

      git diff 5eb3f4c485189f8151d0fe777a31bcfc48adb09e..abf529a33a519c6c712a419b19ac969d20896d6f

      diff --git a/pillar/centrala/production/asdf.sls b/pillar/centrala/production/asdf.sls                                                                                                                                                                  
      index 9ec0e54..bd9f2f6 100644                                                                                                                                                                                                                                 
      --- a/pillar/centrala/production/asdf.sls                                                                                                                                                                                                                  
      +++ b/pillar/centrala/production/asdf.sls                                                                                                                                                                                                                  
      @@ -1,2 +1,9 @@                                                                                                                                                                                                                                               
       # vim: expandtab ts=2 ft=yaml autoindent indentexpr= fdm=marker fmr=/*,*/                                                                                                                                                                                    
                                                                                                                                                                                                                                                                    
      +redukcia:                                                                                                                                                                                                                                                    
      +  db:                                                                                                                                                                                                                                                        
      +    db_bonusy:                                                                                                                                                                                                                                               
      +      dsn: DBI:mysql:bonusy;mysql_socket=/var/lib/mysql/asdf.sock;mysql_read_default_group=db_latin2;mysql_read_default_file=/usr/roft/etc/my.cnf.txt                                                                                                   
      +      user: jankoMrkva                                                                                                                                                                                                                                          
      +      passwd: nourčite                                                                                                                                                                                                                                     
      +                           
      

      PS: I still do not know, why build was triggered at night/morning, because given merge was done at lunch time and so from given time, there should be done plenty of scheduled git pull actions. Also on build time, there were no git commit. It is possible

      Attachments

        Issue Links

          Activity

            lubo_varga_nike Lubo Varga added a comment -

            Found probably issues which are same as my.

            lubo_varga_nike Lubo Varga added a comment - Found probably issues which are same as my.
            lubo_varga_nike Lubo Varga added a comment - - edited

            Trigger of these builds are for sure workspace deletion and than combined with next periodical (each hour) build. Example of pooling log I have sent in issue does have workspace. Todays pooling log triggering build which should not be triggered is this:

            Started on Oct 6, 2016 4:43:00 AM
            No workspace is available, so can’t check for updates.
            Scheduling a new build to get a workspace. (nonexisting_workspace)
            Done. Took 1 ms
            Changes found
            

            So my issue is: Build is triggered when workspace is deleted, despite there is no change to file listed in Included regions between last successful build and current commit. In issue I mention merge commit, this is irrelevant for my case.

            lubo_varga_nike Lubo Varga added a comment - - edited Trigger of these builds are for sure workspace deletion and than combined with next periodical (each hour) build. Example of pooling log I have sent in issue does have workspace. Todays pooling log triggering build which should not be triggered is this: Started on Oct 6, 2016 4:43:00 AM No workspace is available, so can’t check for updates. Scheduling a new build to get a workspace. (nonexisting_workspace) Done. Took 1 ms Changes found So my issue is: Build is triggered when workspace is deleted, despite there is no change to file listed in Included regions between last successful build and current commit . In issue I mention merge commit, this is irrelevant for my case.

            People

              Unassigned Unassigned
              lubo_varga_nike Lubo Varga
              Votes:
              2 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated: