Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-70874

durable-8e1c4e66/script.sh: /bin/sh: bad interpreter: Text file busy

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • durable-task-plugin
    • None
    • CentOS 7
      os.arch amd64
      os.name Linux
      os.version 5.4.238-1.el7.elrepo.x86_64
      java.vendor.version Temurin-17.0.6+10
      java.version 17.0.6
      jenkins.version 2.387.1

      node() {
          stage('1') {
               500.times {
                      sh """#!/bin/sh +e
                          echo ${it}
                      """
                  }
          }    
      }

       

      There will be errors when 60 concurrent executions are performed simultaneously.
      sh: /opt/container/instances/tomcat/app.data/workspace/ddd@53@tmp/durable-4c0dc187/script.sh: /bin/sh: bad interpreter: Text file busy
       

       

       

          [JENKINS-70874] durable-8e1c4e66/script.sh: /bin/sh: bad interpreter: Text file busy

          Mark Waite added a comment - - edited

          I can't duplicate the problem as described. I had to modify the script slightly on my controller because I have Windows agents in addition to Unix agents. The Pipeline that I ran 60 times concurrently without any report of "text file busy" was:

          node('!windows') {
              stage('1') {
                   500.times {
                          sh """#!/bin/sh +e
                              echo ${it}
                          """
                      }
              }    
          }
          

          You'll need to provide more details so that others can duplicate the failure. If those details are not provided, I'll close the issue as "cannot reproduce".

          I was running the plugin versions documented in my plugins.txt file with Jenkins 2.387.2-rc on a container hosted on a Ubuntu 22.04 computer with 50 agents, including a mix of various Unix agents (CentOS 7, Red Hat Enterprise Linux 8, Rocky Linux 9, Debian 10, Debian 11, Debian testing, Debian unstable, Ubuntu 18.04, Ubuntu 20.04, Ubuntu 22.04, FreeBSD 13.1, and OpenBSD 7.2. The agent processor architectures include amd64, aarch64, arm32, and s390x. The controller has 0 executors configured so that all the shell execution is done on the agents. My Unix agents almost all have only a single executor defined. A few of the Unix agents have up to 3 executors. None of them have more than 3 executors.

          Mark Waite added a comment - - edited I can't duplicate the problem as described. I had to modify the script slightly on my controller because I have Windows agents in addition to Unix agents. The Pipeline that I ran 60 times concurrently without any report of "text file busy" was: node('!windows') { stage('1') { 500.times { sh """#!/bin/sh +e echo ${it} """ } } } You'll need to provide more details so that others can duplicate the failure. If those details are not provided, I'll close the issue as "cannot reproduce". I was running the plugin versions documented in my plugins.txt file with Jenkins 2.387.2-rc on a container hosted on a Ubuntu 22.04 computer with 50 agents, including a mix of various Unix agents (CentOS 7, Red Hat Enterprise Linux 8, Rocky Linux 9, Debian 10, Debian 11, Debian testing, Debian unstable, Ubuntu 18.04, Ubuntu 20.04, Ubuntu 22.04, FreeBSD 13.1, and OpenBSD 7.2. The agent processor architectures include amd64, aarch64, arm32, and s390x. The controller has 0 executors configured so that all the shell execution is done on the agents. My Unix agents almost all have only a single executor defined. A few of the Unix agents have up to 3 executors. None of them have more than 3 executors.

          li added a comment -

          I have encountered the same issue on multiple servers with the following hardware configuration: 8-core CPU, 64GB memory, CentOS7 operating system, and kernel version of either 4.18.x or 5.2.x."

          I found that if I change the configuration to the following content, the issue will not occur.

          node() {
              stage('1') {
                   500.times {
                          sh """
                              echo ${it}
                          """
                      }
              }    
          }

          It seems to be related to the "sh" module.

          li added a comment - I have encountered the same issue on multiple servers with the following hardware configuration: 8-core CPU, 64GB memory, CentOS7 operating system, and kernel version of either 4.18.x or 5.2.x." I found that if I change the configuration to the following content, the issue will not occur. node() {     stage('1') {          500.times {                 sh """                     echo ${it}                 """             }     }     } It seems to be related to the "sh" module.

          Markus Winter added a comment -

          How many agents do you have and how many executors each agent has? From above picture you seem at least have enough to run those 60 executions in parallel.

          Markus Winter added a comment - How many agents do you have and how many executors each agent has? From above picture you seem at least have enough to run those 60 executions in parallel.

          li added a comment - - edited

          master configuration

          li added a comment - - edited master configuration

          Mark Waite added a comment - - edited

          The Jenkins documentation on controller isolation says:

          Out of the box, Jenkins is set up to run builds on the built-in node. This is to make it easier to get started with Jenkins, but is inadvisable longer term: Any builds running on the built-in node have the same level of access to the controller file system as the Jenkins process.

          It is therefore highly advisable to not run any builds on the built-in node, instead using agents (statically configured or provided by clouds) to run builds

          CentOS 7 will reach its end of life June 30, 2024. The Platform SIG and the documentation SIG have started discussions to prepare a Jenkins Enhancement Proposal to end Jenkins support of CentOS 7 even before the June 30, 2024 end of life. The meeting notes of the Platform SIG and the Documentation SIG provide examples why CentOS 7 is a poor choice to run Jenkins. Plan your upgrade to another operating system as soon as you reasonably can.

          Mark Waite added a comment - - edited The Jenkins documentation on controller isolation says: Out of the box, Jenkins is set up to run builds on the built-in node. This is to make it easier to get started with Jenkins, but is inadvisable longer term: Any builds running on the built-in node have the same level of access to the controller file system as the Jenkins process. It is therefore highly advisable to not run any builds on the built-in node, instead using agents (statically configured or provided by clouds) to run builds CentOS 7 will reach its end of life June 30, 2024 . The Platform SIG and the documentation SIG have started discussions to prepare a Jenkins Enhancement Proposal to end Jenkins support of CentOS 7 even before the June 30, 2024 end of life. The meeting notes of the Platform SIG and the Documentation SIG provide examples why CentOS 7 is a poor choice to run Jenkins. Plan your upgrade to another operating system as soon as you reasonably can.

          Mark Waite added a comment -

          Jenkins support for Red Hat Enterprise Linux 7 and derivatives like CentOS 7, Scientific Linux 7, and Oracle Linux 7 will end Nov 16, 2023 as announced in the May 2023 blog post. This issue won't be fixed.

          Mark Waite added a comment - Jenkins support for Red Hat Enterprise Linux 7 and derivatives like CentOS 7, Scientific Linux 7, and Oracle Linux 7 will end Nov 16, 2023 as announced in the May 2023 blog post . This issue won't be fixed.

          Christoph Kulla added a comment - - edited

          I see this issue happening occasionally. The output I see is <ws-path>@tmp/durable-e7d2f371/script.sh: Text file busy. It's extremely hard to reproduce, but happens from time to time.

          I believe it's similar to the problem described in JENKINS-48258. The durable-task plugin writes the script text to a temporary file, while another unrelated thread forks a process. The writeable file handle is duplicated in the new process and then the script file can't be executed as Linux prevents executions of writeable files (ETXTBUSY). JENKINS-48258 contains a good analysis and description of the issue.

          A potential fix would be to copy the script file with "cp" and execute the copied file. This is the approach taken here: https://github.com/jenkinsci/git-client-plugin/commit/315ad4e689897dedb20c900b8f32f1ce607fb32c

          Btw, I am not using an outdated CentOS

          Christoph Kulla added a comment - - edited I see this issue happening occasionally. The output I see is <ws-path>@tmp/durable-e7d2f371/script.sh: Text file busy. It's extremely hard to reproduce, but happens from time to time. I believe it's similar to the problem described in JENKINS-48258 . The durable-task plugin writes the script text to a temporary file, while another unrelated thread forks a process. The writeable file handle is duplicated in the new process and then the script file can't be executed as Linux prevents executions of writeable files (ETXTBUSY). JENKINS-48258 contains a good analysis and description of the issue. A potential fix would be to copy the script file with "cp" and execute the copied file. This is the approach taken here: https://github.com/jenkinsci/git-client-plugin/commit/315ad4e689897dedb20c900b8f32f1ce607fb32c Btw, I am not using an outdated CentOS

          Christoph Kulla added a comment - - edited

          I was able to reproduce the problem also with binary wrappers (-Dorg.jenkinsci.plugins.durabletask.BourneShellScript.USE_BINARY_WRAPPER=true):
          common.go:40: fork/exec /var/jenkins_home/<ws-path>@tmp/durable-b5d476b2/script.sh: text file busy
          ERROR: script returned exit code -2

          Christoph Kulla added a comment - - edited I was able to reproduce the problem also with binary wrappers (-Dorg.jenkinsci.plugins.durabletask.BourneShellScript.USE_BINARY_WRAPPER=true): common.go:40: fork/exec /var/jenkins_home/<ws-path>@tmp/durable-b5d476b2/script.sh: text file busy ERROR: script returned exit code -2

            ckullabosch Christoph Kulla
            epoch li
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: