Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-33348

Deleting job under git failed, leaving a poisened commit queue

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Icon: Minor Minor
    • None
    • Jenkins 1.625.3 on Debian Wheezy
      scm-sync-configuration 0.0.9
      git 1.9.1

      After renaming a job, we evidently wound up with a poisoned queue. scm-sync-configuration's attempts to delete the old job were failing:

      [INFO] Executing: /bin/sh -c cd '/var/lib/jenkins/scm-sync-configuration/checkoutConfiguration/jobs' && 'git' 'rm' '-r' '--' 'reset-watchdog-test'
      [INFO] Working directory: /var/lib/jenkins/scm-sync-configuration/checkoutConfiguration/jobs
      Mar 07, 2016 11:09:27 AM hudson.plugins.scm_sync_configuration.SCMManipulator deleteHierarchy
      SEVERE: [deleteHierarchy] Problem during remove : The git command failed.
      

      Status in ~/scm-sync-configuration/checkoutConfiguration was:

      $ git status
      # On branch master
      # Your branch is up-to-date with 'origin/master'.
      #
      # Changes to be committed:
      #   (use "git reset HEAD <file>..." to unstage)
      #
      #	new file:   jobs/reset-watchdog-test/config.xml
      #
      

      The following actions failed:

      1. Manually unstaged the offending (old) config.xml file and its directory
      2. Deleted the local jobs/reset-watchdog-test directory and config.xml file
      3. Committed and pushed.
      4. Made a trivial edit in an unrelated job, and committed it (via the pop-up in the web-app)
      • scm-sync-configuration reported an error
      • The (old) config.xml file had reappeared, and was staged.

      Another approach also failed:

      1. Manually committed the (staged) config.xml file
      2. Made a trivial edit in an unrelated job, and committed it (via the pop-up in the web-app)
      • scm-sync-configuration reported an error

      Examining the tomcat log, scm-sync-configuration was trying to run the following command:

      [INFO] Executing: /bin/sh -c cd '/var/lib/jenkins/scm-sync-configuration/checkoutConfiguration' && 'git' 'add' '--' 'jobs'
      [INFO] Executing: /bin/sh -c cd '/var/lib/jenkins/scm-sync-configuration/checkoutConfiguration' && 'git' 'add' '--' 'jobs/reset-watchdog-test'
      [INFO] Executing: /bin/sh -c cd '/var/lib/jenkins/scm-sync-configuration/checkoutConfiguration' && 'git' 'add' '--' 'jobs/reset-watchdog-test/config.xml'
      [INFO] Executing: /bin/sh -c cd '/var/lib/jenkins/scm-sync-configuration/checkoutConfiguration' && 'git' 'rev-parse' '--show-toplevel'
      [INFO] Executing: /bin/sh -c cd '/var/lib/jenkins/scm-sync-configuration/checkoutConfiguration' && 'git' 'status' '--porcelain' '.'
      [INFO] Executing: /bin/sh -c cd '/var/lib/jenkins/scm-sync-configuration/checkoutConfiguration/jobs' && 'git' 'rm' '-r' '--' 'reset-watchdog-test'
      

      When I ran the git rm -r command myself from the command-line, with the config.xml file staged, git failed:

      error: the following file has changes staged in the index:
          jobs/reset-watchdog-test/config.xml
      (use --cached to keep the file, or -f to force removal)
      

      The workaround that avoided having to restart Jenkins just to un-poison the queue (this is a very busy little box) was to manually commit the rogue config.xml file, so that the git rm -r could work. With the file already staged:

      $ git commit jobs/reset-watchdog-test/config.xml -m"Fake commit"
      [master d348b30] Fake commit
       1 file changed, 103 insertions(+)
       create mode 100644 jobs/reset-watchdog-test/config.xml
      

      I returned to the web-app and made a trivial commit, and the problem was gone. scm-sync-configuration cheerfully deleted the no-longer-wanted job, and pushed it. The plugin's successful commit immediately followed by fake commit in the git log.

      Annoyingly, I just failed to reproduce this by creating a throw-away job and deleting it.

      I originally added this as a comment on JENKINS-25786 in error - that issue was about a rename, in this case we just deleted.

            fcamblor Frédéric Camblor
            douglasr Douglas Royds
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: