Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-37653

Git fetch fails. Returns "Reference directory conflict: refs/heads/" with status code 128

    • Icon: Bug Bug
    • Resolution: Not A Defect
    • Icon: Minor Minor

      I am noticing that the command bellow fails

      git -c core.askpass=true fetch --tags --progress https://github.com/myproject.git +refs/heads/*:refs/remotes/origin/*
      

      While tracing through the error log, I see that it fails to fetch the repo and then returns a status code of 128. What makes this very confusing is that is prints this out as the reason
      stderr: fatal: Reference directory conflict: refs/heads/

          [JENKINS-37653] Git fetch fails. Returns "Reference directory conflict: refs/heads/" with status code 128

          Mark Waite added a comment -

          Considering the large number of installations and users of the git plugin and git client plugin, I suspect this is an issue in your local installation, not in the plugin. That message seems to indicate that command line git found an inconsistency in the .git/refs/heads directory of the workspace.

          You may be able to resolve that complaint by using "Wipe workspace" to delete the entire workspace.

          You may need to check your build process to investigate if it is damaging the contents of the .git directory.

          Mark Waite added a comment - Considering the large number of installations and users of the git plugin and git client plugin, I suspect this is an issue in your local installation, not in the plugin. That message seems to indicate that command line git found an inconsistency in the .git/refs/heads directory of the workspace. You may be able to resolve that complaint by using "Wipe workspace" to delete the entire workspace. You may need to check your build process to investigate if it is damaging the contents of the .git directory.

          Alex Nauda added a comment - - edited

          We are experiencing the same error message, in a GitHub pull request builder job. The first time we saw this was 8/22. We worked around it by removing the entire .git local repo under the job's workspace. The job then ran several times successfully, until it happened again today. We are also hosting our /var/lib/jenkins directory structure on an EFS filesystem. (EFS is Amazon's scalable filesystem product that is exposed to the Jenkins host OS as an NFS mount.)
          Jenkins 2.7.2 (Ubuntu 16.04 package)
          GitHub Pull Request Builder plugin 1.33.0
          Git client 2.7.4 against GitHub (Git 2.6.5)
          Jenkins is not containerized and this job does not involve docker.

          Should we investigate EFS/NFS as a common environment?

          Alex Nauda added a comment - - edited We are experiencing the same error message, in a GitHub pull request builder job. The first time we saw this was 8/22. We worked around it by removing the entire .git local repo under the job's workspace. The job then ran several times successfully, until it happened again today. We are also hosting our /var/lib/jenkins directory structure on an EFS filesystem. (EFS is Amazon's scalable filesystem product that is exposed to the Jenkins host OS as an NFS mount.) Jenkins 2.7.2 (Ubuntu 16.04 package) GitHub Pull Request Builder plugin 1.33.0 Git client 2.7.4 against GitHub (Git 2.6.5) Jenkins is not containerized and this job does not involve docker. Should we investigate EFS/NFS as a common environment?

          Yes, we are also using EFS/NFS.

          Emmanuel Esparza added a comment - Yes, we are also using EFS/NFS.

          Alex Nauda added a comment - - edited

          Git source code location of this error message:
          https://github.com/git/git/blob/master/refs/files-backend.c#L479

          	if ((ref1->flag & REF_DIR) || (ref2->flag & REF_DIR))
          		/* This is impossible by construction */
          		die("Reference directory conflict: %s", ref1->name);
          

          Alex Nauda added a comment - - edited Git source code location of this error message: https://github.com/git/git/blob/master/refs/files-backend.c#L479 if ((ref1->flag & REF_DIR) || (ref2->flag & REF_DIR)) /* This is impossible by construction */ die( "Reference directory conflict: %s" , ref1->name);

          Emmanuel Esparza added a comment - - edited

          Yes, we also saw that and suspect that is caused by a conflicting file left behind from a previous build.

          Emmanuel Esparza added a comment - - edited Yes, we also saw that and suspect that is caused by a conflicting file left behind from a previous build.

          Alex Nauda added a comment -

          Our build doesn't change any tracked files, and the directory is clean:

          $ git status
          HEAD detached at 9b39969
          nothing to commit, working directory clean
          

          Also, the error message and comment imply that something has gone wrong internally in the git repo, doesn't it?

          Alex Nauda added a comment - Our build doesn't change any tracked files, and the directory is clean: $ git status HEAD detached at 9b39969 nothing to commit, working directory clean Also, the error message and comment imply that something has gone wrong internally in the git repo, doesn't it?

          Stav Yagev added a comment -

          My issue is identical but not Jenkins related. I'm posting here since this is the first hit in Google for the above error message.

          I also have a repo hosted on an EFS share mounted via NFS on a regular Ubuntu EC2 instance. For some reason I cannot fetch any more although working directory clean.

          fatal: Reference directory conflict: refs/remotes/origin/
          

          Stav Yagev added a comment - My issue is identical but not Jenkins related. I'm posting here since this is the first hit in Google for the above error message. I also have a repo hosted on an EFS share mounted via NFS on a regular Ubuntu EC2 instance. For some reason I cannot fetch any more although working directory clean. fatal: Reference directory conflict: refs/remotes/origin/

          Mark Waite added a comment -

          There is a wandisco article which describes reasons why a git directory should not be used over an NFS mount. There are also references to poor performance with git repositories hosted over NFS.

          Mark Waite added a comment - There is a wandisco article which describes reasons why a git directory should not be used over an NFS mount. There are also references to poor performance with git repositories hosted over NFS.

          Alex Nauda added a comment -

          I asked this question on the git mailing list.
          markewaite I agree that this looks like it's not a Jenkins bug, but the article you linked does not apply to our use case (a filesystem mounted to a single client over a reliable network). As EFS adoption grows, I expect this use case to become more popular, as it seems like a nice option to manage the growth of the /var/lib/jenkins filesystem.

          Alex Nauda added a comment - I asked this question on the git mailing list. markewaite I agree that this looks like it's not a Jenkins bug, but the article you linked does not apply to our use case (a filesystem mounted to a single client over a reliable network). As EFS adoption grows, I expect this use case to become more popular, as it seems like a nice option to manage the growth of the /var/lib/jenkins filesystem.

            Unassigned Unassigned
            eesparza Emmanuel Esparza
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: