Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-44659

Missing workspace - workspace deleted during concurrent build

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Critical Critical
    • None
    • Jenkins ver. 2.46.1 Linux Ubuntu16.04

      I'm using pipeline to automate builds and it's possible that the same pipeline is triggered concurrently.

      What's happening is I use node ('label'), which already assigned a workspace on a slave with label 'docker_build', then i use ws() to go to another directory as workspace to build.

      Here's what I am doing:

       

      node ('docker_build') {
      currentBuild.description = "${MODEL_NAME}/${BUILD_TYPE} - ${BUILD_LABEL} - Executor ${EXECUTOR_NUMBER} ${NODE_NAME}"
      def actual_workspace = "/home/devops/jenkins_slave_robot/workspace/TinderBox/Chroot_Build/${PROJECT}/${EXECUTOR_NUMBER}"
      ...
      
      stage('Checkout') {
      ws(actual_workspace) {
      ...
      }
      
      stage('Build') {jj
      sh 'whoami'
      def image = docker.image('172.16.181.203:5000/fortios1.0:1.0.0.10')
      image.inside("--privileged -u root -v ${actual_workspace}:${chroot_path}/code") {
      }

      Occasionally,  I would run into issues the missing workspace error. The build is interrupted then failedjj.  The error message looks something like this:

      ERROR: missing workspace /home/devops/jenkins_slave_robot/workspace/TinderBox/FortiOS/Build_Steps/5.4_Chroot_Build on jenkins-smoke-slave03(172.16.182.123)

      From the error message, seems like the original workspace, which was allocated when I use node('label') is disappear. I understand that when the concurrent build happens, the workspace name will come with@<number> , so I can't understand how the workspace would be gone. 

      So far it has happened twice in the last 5 months.

       

          [JENKINS-44659] Missing workspace - workspace deleted during concurrent build

          elhay efrat added a comment -

          this is still happed  i found the RCA

          elhay efrat added a comment - this is still happed  i found the RCA

          Jesse Glick added a comment -

          elhay details?

          Jesse Glick added a comment - elhay details?

          elhay efrat added a comment - - edited

          jglick we can close it 

          the RCA it's deleting the root folder and re-create it 

          i don't see any use case adding it in the code 

          , so what needed to be done , just change the workspace folder to something else and it will work perfectly 

          you need to go to slaves ( node ) > workspace folder , change it , re-run it 

          elhay efrat added a comment - - edited jglick we can close it  the RCA it's deleting the root folder and re-create it  i don't see any use case adding it in the code  , so what needed to be done , just change the workspace folder to something else and it will work perfectly  you need to go to slaves ( node ) > workspace folder , change it , re-run it 

          Jesse Glick added a comment -

          Just correcting the Resolution: Fixed is used for cases where a specific issue was identified and a code change was made to address it.

          Jesse Glick added a comment - Just correcting the Resolution : Fixed is used for cases where a specific issue was identified and a code change was made to address it.

          fan added a comment -

          Why is there a delete directory operation during operation. We also have this problem。

          fan added a comment - Why is there a delete directory operation during operation. We also have this problem。

          phuong added a comment - - edited

          Hi,

          I had same issue with jenkins ver2.389

          Here is what I am doing : 

          • Create jenkins job with pipeline 
            • pipeline {
                  agent none    stages {
                      stage('Docker build & push') {
                          agent { label 'apigateway' }
                          steps {
                              sh '''
                              /kaniko/executor --dockerfile=/home/api-gateway/Dockerfile \
               --context=dir:///home/api-gateway --skip-tls-verify \
               --destination "${ecr_repo}"
                              '''
                          }
                      }
                  }
              }
               
          • The pipeline : we are using kaniko to build docker image on aws ecs
          • But while the job is running, the workspace was deleted

          phuong added a comment - - edited Hi, I had same issue with jenkins ver2.389 Here is what I am doing :  Create jenkins job with pipeline  pipeline {     agent none    stages {         stage( 'Docker build & push' ) {             agent { label 'apigateway' }             steps {                 sh '''                 /kaniko/executor --dockerfile=/home/api-gateway/Dockerfile \  --context=dir: ///home/api-gateway --skip-tls-verify \  --destination "${ecr_repo}"                 '''             }         }     } } The pipeline : we are using kaniko to build docker image on aws ecs But while the job is running, the workspace was deleted

          dracorp added a comment - - edited

          I've identical issue. The problem is not deterministic. So I cannot reproduce it. Sometime, a job is built successfully

          I've also tried to lock workspace with ws() but didn't help.

          I don't use concurrent build for this job.

          dracorp added a comment - - edited I've identical issue. The problem is not deterministic. So I cannot reproduce it. Sometime, a job is built successfully I've also tried to lock workspace with ws() but didn't help. I don't use concurrent build for this job.

          phuong added a comment - - edited

          dracorp you can reproduce this issue when you try build container image with dockerfile has multiple stage

          From my issue : 

          • Dockerfile somethings like this, have two stages : 
          FROM maven:3.8.7-eclipse-temurin-11-alpine AS builder
          WORKDIR /build
          COPY . .
          RUN mvn -fn clean install
          
          FROM .....
          WORKDIR /usr/src/app
          COPY --from=builder /build/target/*.jar app.jar
          ENTRYPOINT ["sh", "-c", "java -jar /usr/src/app/app.jar"] 
          • But, after jenkins run the kaniko command on slave. It can only run stage 1 to build .jar file . After stage 1, the workspace will be deleted by jenkins and throw the error
          • I tried run Dockerfile with separate stage and each stage run successfully 
          • The problem is only when we combine two stages into one Dockerfile

          phuong added a comment - - edited dracorp you can reproduce this issue when you try build container image with dockerfile has multiple stage From my issue :  Dockerfile somethings like this, have two stages :  FROM maven:3.8.7-eclipse-temurin-11-alpine AS builder WORKDIR /build COPY . . RUN mvn -fn clean install FROM ..... WORKDIR /usr/src/app COPY --from=builder /build/target/*.jar app.jar ENTRYPOINT [ "sh" , "-c" , "java -jar /usr/src/app/app.jar" ] But, after jenkins run the kaniko command on slave. It can only run stage 1 to build .jar file . After stage 1, the workspace will be deleted by jenkins and throw the error I tried run Dockerfile with separate stage and each stage run successfully  The problem is only when we combine two stages into one Dockerfile

          dracorp added a comment -

          I've got normal pipeline. In this case, it iterates on AKS, and it gets some metadata from clusters. Nothing special, one job per day.

          ERROR: missing workspace ... on node
          Finished: FAILURE 

          I'm using Jenkins 2.387.1

          dracorp added a comment - I've got normal pipeline. In this case, it iterates on AKS, and it gets some metadata from clusters. Nothing special, one job per day. ERROR: missing workspace ... on node Finished: FAILURE I'm using Jenkins 2.387.1

          Pablo Sanchez added a comment -

          We managed to make it work by changing the remoteFS on the agent configuration

          Pablo Sanchez added a comment - We managed to make it work by changing the remoteFS on the agent configuration

            elhay elhay efrat
            samsun387 Kevin Yu
            Votes:
            0 Vote for this issue
            Watchers:
            13 Start watching this issue

              Created:
              Updated: