Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-70540

Not in a git directory errors

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved (View Workflow)
    • Blocker
    • Resolution: Not A Defect
    • git-plugin

    Description

      Started getting all sorts of checkout errors after upgrading git plugins to the latest version.

       

      Multibranch job indexing started to throw errors for single Gogs repos (Github repos are indexed but fail when running jobs):

       

      Looking up details of XXXXXX...
      Organization URL: XXXXXX
      [Wed Feb 01 13:31:27 GMT 2023] Consulting GitHub Organization
      13:31:27 Connecting to https://api.github.com using Jenkins XXXXXX Looking up repositories of user XXXXXX
        Proposing analysis-model
      Examining Mulgish/analysis-model
      
        Checking branches...
      
        Getting remote branches...
      
          Checking branch master
      No local file defined. Skipping Source Code SCM probe, since Jenkinsfile will be provided by Remote Jenkins File Plugin
          Met criteria
      
        1 branches were processed (query completed)
      
        1 branches were processed
      
      Finished examining Mulgish/analysis-model
      
      Proposing MutinyFX
      Examining Mulgish/MutinyFX
      
        Checking branches...
      
        Getting remote branches...
      
          Checking branch main
      No local file defined. Skipping Source Code SCM probe, since Jenkinsfile will be provided by Remote Jenkins File Plugin
          Met criteria
      
        1 branches were processed (query completed)
      
        1 branches were processed
      
      Finished examining Mulgish/MutinyFX
      
      Proposing warnings-ng-plugin
      Examining Mulgish/warnings-ng-plugin
      
        Checking branches...
      
        Getting remote branches...
      
          Checking branch master
      No local file defined. Skipping Source Code SCM probe, since Jenkinsfile will be provided by Remote Jenkins File Plugin
          Met criteria
      
        1 branches were processed (query completed)
      
        1 branches were processed
      
      Finished examining Mulgish/warnings-ng-plugin
      
      13:31:30 3 repositories were processed
      [Wed Feb 01 13:31:30 GMT 2023] Consulting Single repository
       > git rev-parse --resolve-git-dir /var/jenkins_home/caches/git-e4021de26cf1cebe22fb56083e970129/.git # timeout=10
      Setting origin to https://gogs.internal/xxxxxx
        > git config remote.origin.url https://gogs.internal/xxxxxx 
      # timeout=10
      ERROR: Failed to create or update a subproject xxxxxx
      hudson.plugins.git.GitException: Command "git config remote.origin.url https://gogs.internal/xxxxxx" returned status code 128:
      stdout: 
      stderr: fatal: not in a git directory
      
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2734)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2660)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2656)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1981)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1993)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1601)
      	at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:161)
      	at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:381)
      	at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:350)
      	at jenkins.plugins.git.AbstractGitSCMSource.retrieve(AbstractGitSCMSource.java:590)
      	at jenkins.scm.api.SCMSource._retrieve(SCMSource.java:373)
      	at jenkins.scm.api.SCMSource.fetch(SCMSource.java:327)
      	at jenkins.branch.MultiBranchProjectFactory$BySCMSourceCriteria.recognizes(MultiBranchProjectFactory.java:261)
      	at jenkins.branch.OrganizationFolder$SCMSourceObserverImpl$1.recognizes(OrganizationFolder.java:1358)
      	at jenkins.branch.OrganizationFolder$SCMSourceObserverImpl$1.complete(OrganizationFolder.java:1373)
      	at jenkins.scm.impl.SingleSCMNavigator.visitSources(SingleSCMNavigator.java:78)
      	at jenkins.branch.OrganizationFolder.computeChildren(OrganizationFolder.java:535)
      	at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:278)
      	at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:166)
      	at jenkins.branch.OrganizationFolder$OrganizationScan.run(OrganizationFolder.java:917)
      	at hudson.model.ResourceController.execute(ResourceController.java:101)
      	at hudson.model.Executor.run(Executor.java:442)
      [Wed Feb 01 13:31:30 GMT 2023] Consulting Single repository
       > git rev-parse --resolve-git-dir /var/jenkins_home/caches/git-992dbde355d1d72f7a50c0b5f51d0e23/.git # timeout=10
      Setting origin to https://gogs.internal/xxxxxx  > git config remote.origin.url https://gogs.internal/xxxxxx # timeout=10
      ERROR: Failed to create or update a subproject xxxxxx
      hudson.plugins.git.GitException: Command "git config remote.origin.url https://gogs.internal/xxxxxx" returned status code 128:
      stdout: 
      stderr: fatal: not in a git directory
      
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2734)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2660)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2656)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1981)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1993)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1601)
      	at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:161)
      	at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:381)
      	at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:350)
      	at jenkins.plugins.git.AbstractGitSCMSource.retrieve(AbstractGitSCMSource.java:590)
      	at jenkins.scm.api.SCMSource._retrieve(SCMSource.java:373)
      	at jenkins.scm.api.SCMSource.fetch(SCMSource.java:327)
      	at jenkins.branch.MultiBranchProjectFactory$BySCMSourceCriteria.recognizes(MultiBranchProjectFactory.java:261)
      	at jenkins.branch.OrganizationFolder$SCMSourceObserverImpl$1.recognizes(OrganizationFolder.java:1358)
      	at jenkins.branch.OrganizationFolder$SCMSourceObserverImpl$1.complete(OrganizationFolder.java:1373)
      	at jenkins.scm.impl.SingleSCMNavigator.visitSources(SingleSCMNavigator.java:78)
      	at jenkins.branch.OrganizationFolder.computeChildren(OrganizationFolder.java:535)
      	at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:278)
      	at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:166)
      	at jenkins.branch.OrganizationFolder$OrganizationScan.run(OrganizationFolder.java:917)
      	at hudson.model.ResourceController.execute(ResourceController.java:101)
      	at hudson.model.Executor.run(Executor.java:442) 

       

       

       

      Pipelines started failing with below errors:

       

      22:50:45  The recommended git tool is: NONE
      22:50:45  using credential jenkins_svc_vcs
      22:50:45   > git rev-parse --resolve-git-dir /var/jenkins_home/workspace/CI_foobar_main@script/a06cb9a0b5c86ab8e2e832b21c314405ffcb0a9520a6c56990348fa4eb7a901f/.git # timeout=10
      22:50:45  Fetching changes from the remote Git repository
      22:50:45   > git config remote.origin.url https://gogs.internal/xxxx # timeout=10
      22:50:45  ERROR: Error fetching remote repo 'origin'
      22:50:45  hudson.plugins.git.GitException: Failed to fetch from https://gogs.internal/xxxx 22:50:45      at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:1003)
      22:50:45      at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1245)
      22:50:45      at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1309)
      22:50:45      at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:129)
      22:50:45      at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:159)
      22:50:45      at org.jenkinsci.plugins.workflow.multibranch.extended.scm.ExtendedSCMBinder.create(ExtendedSCMBinder.java:120)
      22:50:45      at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:312)
      22:50:45      at hudson.model.ResourceController.execute(ResourceController.java:101)
      22:50:45      at hudson.model.Executor.run(Executor.java:442)
      22:50:45  Caused by: hudson.plugins.git.GitException: Command "git config remote.origin.url https://gogs.internal/xxxx" returned status code 128:
      22:50:45  stdout: 
      22:50:45  stderr: fatal: not in a git directory
      22:50:45  
      22:50:45      at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2734)
      22:50:45      at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2660)
      22:50:45      at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2656)
      22:50:45      at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1981)
      22:50:45      at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1993)
      22:50:45      at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1601)
      22:50:45      at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:161)
      22:50:45      at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:991)
      22:50:45      ... 8 more
      22:50:45  ERROR: Error fetching remote repo 'origin'
      22:50:45  ERROR: Maximum checkout retry attempts reached, aborting 

       

       

      Attachments

        1. jobs-freestyle-config.xml
          1 kB
        2. plugins.txt
          0.8 kB
        3. run-jenkins.sh
          1 kB

        Activity

          markewaite Mark Waite added a comment -

          mulgish I've not yet seen a description that details how to duplicate the problem. Will you be able to provide more details?

          markewaite Mark Waite added a comment - mulgish I've not yet seen a description that details how to duplicate the problem. Will you be able to provide more details?
          openpablo Pablo added a comment - - edited

          Updating these plugins:

          -apache-httpcomponents-client-4-api:4.5.13-1.0
          +apache-httpcomponents-client-4-api:4.5.13-138.v4e7d9a_7b_a_e61

          -jackson2-api:2.14.1-313.v504cdd45c18b
          +jackson2-api:2.14.2-319.v37853346a_229

          With jenkins version: jenkins/jenkins:2.375.2-lts-jdk11

          Jenkinsfile content doesn't matter, fails before anything can execute.

           

          openpablo Pablo added a comment - - edited Updating these plugins: -apache-httpcomponents-client-4-api:4.5.13-1.0 + apache-httpcomponents-client-4-api:4.5.13-138.v4e7d9a_7b_a_e61 -jackson2-api:2.14.1-313.v504cdd45c18b + jackson2-api:2.14.2-319.v37853346a_229 With jenkins version: jenkins/jenkins : 2.375.2-lts-jdk11 Jenkinsfile content doesn't matter, fails before anything can execute.  
          markewaite Mark Waite added a comment -

          openpablo I use those same plugin versions in my installation without any issue. I suspect there is something else happening in your job definition that causes the behavior on your system and not on mine. Is your Jenkinsfile performing multiple calls to checkout scm or to git?

          markewaite Mark Waite added a comment - openpablo I use those same plugin versions in my installation without any issue. I suspect there is something else happening in your job definition that causes the behavior on your system and not on mine. Is your Jenkinsfile performing multiple calls to checkout scm or to git ?
          mulgish Juri Duval added a comment - - edited

          markewaite I am struggling to reproduce this issue with fresh installation.

          It seems like the issue occurs when upgrading from older version of jenkins and plugins. 

          It takes a lot of time to manually install old versions of the plugins, so I did not spend time trying that (can probably install old plugins in custom dockerfile).

           

           

          On a positive note, I managed to solve the issue by rebuilding Jenkins Dockerfile to have uid/guid matching to that of my NFS share.

          Confusing 'not a git directory' simply disappeared.

           

          Please find steps I tried to reproduce the issue (without an nfs share but should be similar behaviour):

          1. Create jenkins home dir via 
            mkdir jenkins_home
            
          1. Update directory uid and permissions (Jenkins expects 1000 but has rw access via "other" permissions)
            chown 1010:1010 jenkins_home
            sudo chmod uga+rwx jenkins_home
          2. Create jenkins.yaml compose file with below contents
            version: "3.9"
            services:
              controller:
                image: jenkins/jenkins:latest
                environment:
                  TZ: 'Europe/London'
                  JAVA_OPTS: '-Xms4g -Xmx4g -XX:+AlwaysPreTouch'
                ports:
                  - "8080:8080"
                networks:
                  - "jenkins-net"
                volumes:
                  - type: bind
                    source: ./jenkins_home
                    target: /var/jenkins_home
                deploy:
                  mode: replicated
                  replicas: 1
                  resources:
                    limits:
                      cpus: '3.000'
                      memory: 8G
                    reservations:
                      cpus: '0.500'
                      memory: 4G
            networks:
              jenkins-net: 
          3. Spin up jenkins via Docker or Podman compose  
            podman-compose --file=jenkins.yaml -p jenkins up
            docker-compose --file=jenkins.yaml -p jenkins up 
          4. Access jenkins via http://localhost:8080 and go through default installation process (with recommended plugins)
          5. Create a multibranch pipeline and point it to https://github.com/jenkinsci/git-plugin
          6. Observe no "not a git directory" errors. Strangely git also does not care that uid and gid are mismatching.
            Started by user admin
            [Tue Feb 21 00:53:19 GMT 2023] Starting branch indexing...
             > git --version # timeout=10
             > git --version # 'git version 2.30.2'
             > git ls-remote --symref -- https://github.com/jenkinsci/git-plugin # timeout=10
             > git rev-parse --resolve-git-dir /var/jenkins_home/caches/git-5f7e7d9a6699d86ffef71261765a10ab/.git # timeout=10
            Setting origin to https://github.com/jenkinsci/git-plugin
             > git config remote.origin.url https://github.com/jenkinsci/git-plugin # timeout=10
            Fetching & pruning origin...
            Listing remote references...
             > git config --get remote.origin.url # timeout=10
             > git --version # timeout=10
             > git --version # 'git version 2.30.2'
             > git ls-remote -h -- https://github.com/jenkinsci/git-plugin # timeout=10
            Fetching upstream changes from origin
             > git config --get remote.origin.url # timeout=10
             > git fetch --tags --force --progress --prune -- origin +refs/heads/*:refs/remotes/origin/* # timeout=10
            Checking branches...
              Checking branch 2.6.x
                  ‘Jenkinsfile’ not found
                Does not meet criteria
              Checking branch v3.10.0.x 

            Not sure if it is worth getting to the root cause of those errors given that the issue can be solved by matching uid/gid (it is best practice anyways).

           

          mulgish Juri Duval added a comment - - edited markewaite I am struggling to reproduce this issue with fresh installation. It seems like the issue occurs when upgrading from older version of jenkins and plugins.  It takes a lot of time to manually install old versions of the plugins, so I did not spend time trying that (can probably install old plugins in custom dockerfile).     On a positive note, I managed to solve the issue by rebuilding Jenkins Dockerfile to have uid/guid matching to that of my NFS share. Confusing 'not a git directory' simply disappeared.   Please find steps I tried to reproduce the issue (without an nfs share but should be similar behaviour): Create jenkins home dir via  mkdir jenkins_home Update directory uid and permissions (Jenkins expects 1000 but has rw access via "other" permissions) chown 1010:1010 jenkins_home sudo chmod uga+rwx jenkins_home Create jenkins.yaml compose file with below contents version: "3.9" services:   controller:     image: jenkins/jenkins:latest     environment:       TZ: 'Europe/London'       JAVA_OPTS: '-Xms4g -Xmx4g -XX:+AlwaysPreTouch'     ports:       - "8080:8080"     networks:       - "jenkins-net"     volumes:       - type: bind         source: ./jenkins_home         target: / var /jenkins_home     deploy:       mode: replicated       replicas: 1       resources:         limits:           cpus: '3.000'           memory: 8G         reservations:           cpus: '0.500'           memory: 4G networks:   jenkins-net: Spin up jenkins via Docker or Podman compose   podman-compose --file=jenkins.yaml -p jenkins up docker-compose --file=jenkins.yaml -p jenkins up Access jenkins via http://localhost:8080 and go through default installation process (with recommended plugins) Create a multibranch pipeline and point it to https://github.com/jenkinsci/git-plugin Observe no "not a git directory" errors. Strangely git also does not care that uid and gid are mismatching. Started by user admin [Tue Feb 21 00:53:19 GMT 2023] Starting branch indexing... > git --version # timeout=10 > git --version # 'git version 2.30.2' > git ls-remote --symref -- https: //github.com/jenkinsci/git-plugin # timeout=10 > git rev-parse --resolve-git-dir / var /jenkins_home/caches/git-5f7e7d9a6699d86ffef71261765a10ab/.git # timeout=10 Setting origin to https: //github.com/jenkinsci/git-plugin > git config remote.origin.url https: //github.com/jenkinsci/git-plugin # timeout=10 Fetching & pruning origin... Listing remote references... > git config --get remote.origin.url # timeout=10 > git --version # timeout=10 > git --version # 'git version 2.30.2' > git ls-remote -h -- https: //github.com/jenkinsci/git-plugin # timeout=10 Fetching upstream changes from origin > git config --get remote.origin.url # timeout=10 > git fetch --tags --force --progress --prune -- origin +refs/heads/*:refs/remotes/origin/* # timeout=10 Checking branches... Checking branch 2.6.x ‘Jenkinsfile’ not found Does not meet criteria Checking branch v3.10.0.x Not sure if it is worth getting to the root cause of those errors given that the issue can be solved by matching uid/gid (it is best practice anyways).  
          markewaite Mark Waite added a comment -

          Based on your comments about ownership of the directory, I can duplicate the same conditions with current plugins. I believe the key step is that the directory with incorrect ownership needs to already exist and contain a git repository.

          • Place the plugins.txt and run-jenkins.sh files in an empty directory
          • Create the jobs/freestyle directory and copy jobs-freestyle-config.xml as jobs/freestyle/config.xml
          • Perform an sudo command so that sudo will cache the authentication
          • Run the run-jenkins.sh script (it contains an sudo command to change ownership of a cloned repository
          • Provide the generated password, create a user, and answer the installation wizard by installing no plugins (already installed)
          • Run the job name freestyle and watch it fail with not in a git directory

          Based on that, I'm closing this issue as not a defect. Command line git has decided that it is dangerous to perform git operations in a directory owned by a different user. I may consider a way to better detect it and better alert the administrator that there is a problem, but ultimately it is a mistake to perform git operations in a directory owned by a different user without intentionally configuring the safe.directory to allow it.

          markewaite Mark Waite added a comment - Based on your comments about ownership of the directory, I can duplicate the same conditions with current plugins. I believe the key step is that the directory with incorrect ownership needs to already exist and contain a git repository. Place the plugins.txt and run-jenkins.sh files in an empty directory Create the jobs/freestyle directory and copy jobs-freestyle-config.xml as jobs/freestyle/config.xml Perform an sudo command so that sudo will cache the authentication Run the run-jenkins.sh script (it contains an sudo command to change ownership of a cloned repository Provide the generated password, create a user, and answer the installation wizard by installing no plugins (already installed) Run the job name freestyle and watch it fail with not in a git directory Based on that, I'm closing this issue as not a defect. Command line git has decided that it is dangerous to perform git operations in a directory owned by a different user. I may consider a way to better detect it and better alert the administrator that there is a problem, but ultimately it is a mistake to perform git operations in a directory owned by a different user without intentionally configuring the safe.directory to allow it.

          People

            Unassigned Unassigned
            mulgish Juri Duval
            Votes:
            2 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: