Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-49759

Can't create temp file in Kubernetes Plugin Pipeline

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • kubernetes-plugin
    • Kuberetes 1.8
      Jenkins 2.107
      Kubernetes Plugin: 1.2.1
      Java Version: 1.8.0_151
      Jenkins Master running in Docker Container in k8s

      We are having an issue where we can not create a temp file in a build pipeline within a PodTemplate. Oddly inserting a random 'sh' command before the creation of the temp file solved the problem. Here is the code that fails:

       

      import hudson.FilePath;
      import jenkins.model.Jenkins;
      
      def label = "mynode"
      podTemplate(
        label: label,
        nodeSelector: "kops.k8s.io/instancegroup=jenkins-workers",
        containers: [containerTemplate(name: 'kubectl', image: 'lachlanevenson/k8s-kubectl:v1.9.3', command: 'cat', ttyEnabled: true)]) {
      
        node(label) {
          container("kubectl") {
            def workspace = new FilePath(Jenkins.getInstance().getComputer(env.NODE_NAME).getChannel(), env.WORKSPACE);
            workspace.createTempFile("test", ".txt")
            sh "ls -al"
          }
        }
      }
      

       

      The full log is attached, but essentially we get the following exception:

      Caused: java.io.IOException: Failed to create a temp file on /home/jenkins/workspace/folder/test-job
      at hudson.FilePath.createTempFile(FilePath.java:1330)
      

      Oddly, if we add a "sh" command before the creation of the temp file the job succeeds. The following code works:

      import hudson.FilePath;
      import jenkins.model.Jenkins;
      
      def label = "mynode"
      podTemplate(
        label: label,
        nodeSelector: "kops.k8s.io/instancegroup=jenkins-workers",
        containers: [containerTemplate(name: 'kubectl', image: 'lachlanevenson/k8s-kubectl:v1.9.3', command: 'cat', ttyEnabled: true)]) {
      
        node(label) {
          container("kubectl") {
            sh "ls -al" // This added code somehow makes this pipeline succeed.
            def workspace = new FilePath(Jenkins.getInstance().getComputer(env.NODE_NAME).getChannel(), env.WORKSPACE);
            workspace.createTempFile("test", ".txt")
            sh "ls -al"
          }
        }
      }
      

      It does not seem to matter what the "sh" command before the temp file creation is. We have used "pwd", "echo $HOME", etc. and they all have the same effect of making the pipeline work. Of course both sh "ls -al" commands were put in there as debugging steps, and it just so happened that the first one accidentally made the pipeline work.

       

          [JENKINS-49759] Can't create temp file in Kubernetes Plugin Pipeline

          I'm actually quite surprised it works at all.
          From my understanding createTempFile runs on the agent container, and because the filesystem is shared you could see the file in all the containers. So it would work when outside the container step.

          Inside container you can execute commands, but that's all you can do because it uses kubectl exec

          Carlos Sanchez added a comment - I'm actually quite surprised it works at all. From my understanding createTempFile runs on the agent container, and because the filesystem is shared you could see the file in all the containers. So it would work when outside the container step. Inside container you can execute commands, but that's all you can do because it uses kubectl exec

          Michael MacFadden added a comment - - edited

          csanchez Thanks for your comments, and thanks for looking at this. You'll have to forgive my analysis here since I am fairly new at looking through the Jenkins Code Base and understanding how it works under the hood.

          The behavior is the same regardless of if you put it inside the "container" block or not. It works with the added "sh" command and does not work without it. So it really doesn't' matter if you put it in the container, or directly in the node. Like so:

          podTemplate(
            label: label,
            nodeSelector: "kops.k8s.io/instancegroup=jenkins-workers",
            containers: [containerTemplate(name: 'kubectl', image: 'lachlanevenson/k8s-kubectl:v1.9.3', command: 'cat', ttyEnabled: true)]) {
          
            node(label) {
              sh "ls -al" // Works with this, does not work without it.
              def workspace = new FilePath(Jenkins
                .getInstance()
                .getComputer(env.NODE_NAME)
                .getChannel(), 
                env.WORKSPACE);
              workspace.createTempFile("test", ".txt")
              sh "ls -al"
          
              container("kubectl") {
                // some commands inside the container.
              }
            }
          }
          

          I believe that pipeline steps are interpreted on the master and the sent to the slaves . So when you call createTempFile, the master is interpreting that request and ultimately asks the salve to create the file. This salve happens to be the JNLP slave running in a container itself. The point being "createTempFile" is not running "inside" the container, no matter where it appears in the pipeline code block. It's actually the master that interprets it and then asks the slave to create the file. In the case of the createTempFile, it's not going to actually ask the container to do it, but rather the JNLP Slave container running of the JNLP protocol directly to that container (not using kubectl exec).

          You are correct, then that the filesystem is shared between the JNLP Slave and the kubectl container, so the temp file should then be available inside the kubectl container.

          So all that said, I think the issue is still valid.

           

          Michael MacFadden added a comment - - edited csanchez Thanks for your comments, and thanks for looking at this. You'll have to forgive my analysis here since I am fairly new at looking through the Jenkins Code Base and understanding how it works under the hood. The behavior is the same regardless of if you put it inside the "container" block or not. It works with the added "sh" command and does not work without it. So it really doesn't' matter if you put it in the container, or directly in the node. Like so: podTemplate( label: label, nodeSelector: "kops.k8s.io/instancegroup=jenkins-workers" , containers: [containerTemplate(name: 'kubectl' , image: 'lachlanevenson/k8s-kubectl:v1.9.3' , command: 'cat' , ttyEnabled: true )]) { node(label) { sh "ls -al" // Works with this , does not work without it. def workspace = new FilePath(Jenkins .getInstance() .getComputer(env.NODE_NAME) .getChannel(), env.WORKSPACE); workspace.createTempFile( "test" , ".txt" ) sh "ls -al" container( "kubectl" ) { // some commands inside the container. } } } I believe that pipeline steps are interpreted on the master and the sent to the slaves . So when you call createTempFile, the master is interpreting that request and ultimately asks the salve to create the file. This salve happens to be the JNLP slave running in a container itself. The point being "createTempFile" is not running "inside" the container, no matter where it appears in the pipeline code block. It's actually the master that interprets it and then asks the slave to create the file. In the case of the createTempFile, it's not going to actually ask the container to do it, but rather the JNLP Slave container running of the JNLP protocol directly to that container (not using kubectl exec). You are correct, then that the filesystem is shared between the JNLP Slave and the kubectl container, so the temp file should then be available inside the kubectl container. So all that said, I think the issue is still valid.  

            Unassigned Unassigned
            mmacfadden Michael MacFadden
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: