Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-48258

git client plugin occasionally fails with "text file busy" error

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Minor Minor
    • git-client-plugin
    • None
    • ubuntu 16.04

      Occasionally I get the following git error early in some of my pipeline builds, but have not been able to reproduce the issue during testing.

      Perhaps the createUnixGitSSH method in CliGitAPIImpl.java should explicitly close the PrintWriter before returning?  Or maybe it just needs to flush the stream, and the script should be executed as "sh tmpfile.sh" instead of "./tmpfile.sh" ?

      using GIT_SSH to set credentials 
       > git ls-remote -h -t git.example.org:/var/git/foo.git # timeout=10
      hudson.plugins.git.GitException: Command "git ls-remote -h -t git.example.org:/var/git/foo.git" returned status code 128:
      stdout: 
      stderr: fatal: cannot exec '/tmp/ssh4044675960910064496.sh': Text file busy
      fatal: unable to fork
      
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1970)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1689)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1600)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1591)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.getRemoteReferences(CliGitAPIImpl.java:2785)
      	at jenkins.plugins.git.AbstractGitSCMSource.retrieve(AbstractGitSCMSource.java:708)
      	at jenkins.scm.api.SCMSource.fetch(SCMSource.java:598)
      	at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrieve(SCMSourceRetriever.java:80)
      	at org.jenkinsci.plugins.workflow.libs.LibraryAdder.retrieve(LibraryAdder.java:153)
      	at org.jenkinsci.plugins.workflow.libs.LibraryAdder.add(LibraryAdder.java:134)
      	at org.jenkinsci.plugins.workflow.libs.LibraryDecorator$1.call(LibraryDecorator.java:125)
      	at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1065)
      	at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603)
      	at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581)
      	at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558)
      	at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
      	at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
      	at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688)
      	at groovy.lang.GroovyShell.parse(GroovyShell.java:700)
      	at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:129)
      	at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:123)
      	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:517)
      	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:480)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:269)
      	at hudson.model.ResourceController.execute(ResourceController.java:97)
      	at hudson.model.Executor.run(Executor.java:421)
      org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
      WorkflowScript: Loading libraries failed
      
      1 error
      

          [JENKINS-48258] git client plugin occasionally fails with "text file busy" error

          Pavel Novak added a comment - - edited

          markewaite 

          Hello, Mark, 
          I apologize but I can't afford to install beta version.

          Anyway thanks for the feedback about the update.

          I noticed it looks like solution was found, good job.

           

          From my experience, I did not found any relation between environment, we are running on RHEL and tried with jenkins LTS 2.107.3 and 2.150.2, issue occurs on both machines, randomly and not so frequently.
          I am guessing if its not having eg. some relation to the credentials which are being used, but I am not sure (eg. stored on the global level and then being used in parallel)

          Pavel Novak added a comment - - edited markewaite   Hello, Mark,  I apologize but I can't afford to install beta version. Anyway thanks for the feedback about the update. I noticed it looks like solution was found, good job.   From my experience, I did not found any relation between environment, we are running on RHEL and tried with jenkins LTS 2.107.3 and 2.150.2, issue occurs on both machines, randomly and not so frequently. I am guessing if its not having eg. some relation to the credentials which are being used, but I am not sure (eg. stored on the global level and then being used in parallel)

          joe ferr added a comment -

          Way above in this thread I mentioned that I compiled/deployed from the Pull Request and it fixed the issue.  We have a large number of parallel gitlab activity going on in this jenkins instance....lots of pipeline builds to support many microservices and most jobs retrieve from multiple git repos.

          Just wanted to let you know that someone "upgraded" our instance to the latest released version and as-expected the problem came back.

          I installed the latest beta (4) a few days back and everything is great again....just another confirmation real-world that the fix works fine.

          Thanks,

          Joe

          joe ferr added a comment - Way above in this thread I mentioned that I compiled/deployed from the Pull Request and it fixed the issue.  We have a large number of parallel gitlab activity going on in this jenkins instance....lots of pipeline builds to support many microservices and most jobs retrieve from multiple git repos. Just wanted to let you know that someone "upgraded" our instance to the latest released version and as-expected the problem came back. I installed the latest beta (4) a few days back and everything is great again....just another confirmation real-world that the fix works fine. Thanks, Joe

          Mark Waite added a comment -

          Thanks very much joeferr!

          Mark Waite added a comment - Thanks very much joeferr !

          joe ferr added a comment -

          markewaite After installing beta4 our gitlab integration from jenkins to gitlab stopped working.  There seem to be lots of reported issues w. the gitlab plugin so I'm not sure what broke it (e.g may be unrelated to your stuff)...here's a ticket that we found

          https://github.com/jenkinsci/gitlab-plugin/issues/893.

          We have hundreds of pipeline jobs that use this and they all stopped reporting pipeline status back to gitlab.

           

          I turned on logging for the gitlab plugin and I'm seeing this.

          If you do think it's related to  your changes let me know if I can do any testing for you to  help troubleshoot.

           

          Build does not contain build data.
          Feb 05, 2019 9:46:44 AM INFO com.dabsquared.gitlabjenkins.util.CommitStatusUpdater retrieveGitlabProjectIds
           

          joe ferr added a comment - markewaite After installing beta4 our gitlab integration from jenkins to gitlab stopped working.  There seem to be lots of reported issues w. the gitlab plugin so I'm not sure what broke it (e.g may be unrelated to your stuff)...here's a ticket that we found https://github.com/jenkinsci/gitlab-plugin/issues/893. We have hundreds of pipeline jobs that use this and they all stopped reporting pipeline status back to gitlab.   I turned on logging for the gitlab plugin and I'm seeing this. If you do think it's related to  your changes let me know if I can do any testing for you to  help troubleshoot.   Build does not contain build data. Feb 05, 2019 9:46:44 AM INFO com.dabsquared.gitlabjenkins.util.CommitStatusUpdater retrieveGitlabProjectIds  

          Pavel Novak added a comment -

          joeferr

          Yes, I can confirm that, once we upgraded to the latest ( git plugin to 4.0.0-rc and git client plugin to 3.0.0-rc), git operations were not working correctly. 

          Downgrade to 3.9.1 and 2.7.2 solved the issue, but it seems (of course) busy file persist.

          Pavel Novak added a comment - joeferr Yes, I can confirm that, once we upgraded to the latest ( git plugin to 4.0.0-rc and git client plugin to 3.0.0-rc), git operations were not working correctly.  Downgrade to 3.9.1 and 2.7.2 solved the issue, but it seems (of course) busy file persist.

          Sebastian Wojas added a comment - - edited

          markewaite when you're planning to release fixed version (3.0, not beta) of git-plugin ?

          Sebastian Wojas added a comment - - edited markewaite when you're planning to release fixed version (3.0, not beta) of git-plugin ?

          Mark Waite added a comment - - edited

          sebawo when I've resolved the blocking bugs that are identified in the git client plugin 3.0 milestone and in the git plugin 4.0 milestone. You're welcome to help test the beta versions and report your results of testing the beta versions.

          Mark Waite added a comment - - edited sebawo when I've resolved the blocking bugs that are identified in the git client plugin 3.0 milestone and in the git plugin 4.0 milestone . You're welcome to help test the beta versions and report your results of testing the beta versions.

          Mark Waite added a comment -

          Released with git client plugin 3.0.0 and git plugin 4.0.0 on Nov 2, 2019.

          Mark Waite added a comment - Released with git client plugin 3.0.0 and git plugin 4.0.0 on Nov 2, 2019.

          ASHOK MOHANTY added a comment -

          Can I only upgrade git-client plugin to v3.0.0 (right now we have git plugin v3.9.1) !? OR we should upgrade git client plugin 3.0.0 and git plugin 4.0.0 together !!

          ASHOK MOHANTY added a comment - Can I only upgrade git-client plugin to v3.0.0 (right now we have git plugin v3.9.1) !? OR we should upgrade  git client plugin 3.0.0 and git plugin 4.0.0 together !!

          Mark Waite added a comment -

          Upgrade the two plugins together. Git plugin 3.x might work with git client plugin 3.0.0, but you'll be the first (and possibly only) person to test that configuration. It is intended that git plugin 4.0.0 and git client plugin 3.0.0 upgrade together.

          It is much better to travel with large groups of people in this case. Large groups of people have tested git client plugin 3.0.0 with git plugin 4.0.0. I'm not aware of anyone that has tested git client plugin 3.0.0 with git plugin 3.x releases.

          Mark Waite added a comment - Upgrade the two plugins together. Git plugin 3.x might work with git client plugin 3.0.0, but you'll be the first (and possibly only) person to test that configuration. It is intended that git plugin 4.0.0 and git client plugin 3.0.0 upgrade together. It is much better to travel with large groups of people in this case. Large groups of people have tested git client plugin 3.0.0 with git plugin 4.0.0. I'm not aware of anyone that has tested git client plugin 3.0.0 with git plugin 3.x releases.

            Unassigned Unassigned
            mbmop Mostyn Bramley-Moore
            Votes:
            8 Vote for this issue
            Watchers:
            27 Start watching this issue

              Created:
              Updated:
              Resolved: