Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-12699

Temp upload files from file param not removed after transferred to slave

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • core
    • OS: Linux ara02 2.6.9-103.ELsmp #1 SMP Fri Nov 11 14:34:02 EST 2011 i686 i686 i386 GNU/Linux
      JRE: java version "1.6.0_16"
      Jenkins: 1.442

      I currently have a master-slave configuration on Jenkins 1.442 running on a Linux box. Lately I started using file parameter for one of my jobs, and noticed that disk space was decreasing rather quickly on the master system. Upon examination it turns out that file parameter is creating files under /tmp, with a naming convention upload_*.tmp, when the file parameters are uploaded. I've done some experiment and noticed that:

      • If the job is running on master, the tmp file is moved into the workspace correctly.
      • If the job is running on slave, the tmp file is uploaded to the slave workspace correctly, but the upload_*.tmp file remains in /tmp.

      I am using a file parameter for a 100 MB file, so it is very easy to collect GBs worth of temporary files, until my master runs out of disk space. For now I have added a cron job to delete these upload_*.tmp files, but it's not safe if the script runs at the same time as a user uploading a file param for a job.

      Let me know what other debugging information I need to provide, if any. Please investigate, thanks.

          [JENKINS-12699] Temp upload files from file param not removed after transferred to slave

          evernat added a comment -

          Is it reproduced with a recent Jenkins version?

          evernat added a comment - Is it reproduced with a recent Jenkins version?

          evernat added a comment -

          No response from the reporter, so resolving as incomplete.

          evernat added a comment - No response from the reporter, so resolving as incomplete.

          Reopening as per JENKINS-29121.

          Christopher Orr added a comment - Reopening as per JENKINS-29121 .

          I'm not seeing this behaviour with large file parameters, whether the build is executed on master or a slave.

          But I'm not using buildWithParameters, but rather the build format documented here:
          https://wiki.jenkins-ci.org/display/JENKINS/Remote+access+API#RemoteaccessAPI-Submittingjobs

          i.e.

          curl -v -X POST http://jenkins.local:8080/job/upload-test/build \
            --form f0=@/data/fw123.bin \
            --form json='{"parameter": [{"name":"firmware.bin", "file":"f0"}]}'
          

          When the job is running, I see "Copying file to firmware.bin" in the console output.
          But checking /tmp and the java.io.tmpdir directory on the slave while this is happening shows no temporary files being created.

          toshee2000: Can you provide more detail on exactly how you're starting a job?

          Christopher Orr added a comment - I'm not seeing this behaviour with large file parameters, whether the build is executed on master or a slave. But I'm not using buildWithParameters , but rather the build format documented here: https://wiki.jenkins-ci.org/display/JENKINS/Remote+access+API#RemoteaccessAPI-Submittingjobs i.e. curl -v -X POST http://jenkins.local:8080/job/upload-test/build \ --form f0=@/data/fw123.bin \ --form json='{"parameter": [{"name":"firmware.bin", "file":"f0"}]}' When the job is running, I see "Copying file to firmware.bin" in the console output. But checking /tmp and the java.io.tmpdir directory on the slave while this is happening shows no temporary files being created. toshee2000 : Can you provide more detail on exactly how you're starting a job?

          Elgin Peteza added a comment - - edited

          Basically, I execute a job by POSTing to Jenkins using python's requests module via a remote PC.
          i.e.

          import requests
          with open("file.zip", "rb") as z:
              file_to_send = {"jenkins_file.zip": z}
              requests.post("http://jenkins.local:8080/job/my_job/buildWithParameters", files=file_to_send)
          

          An upload_*.tmp file is created in /tmp directory of the master after executing the script. The master then builds the job on one of its nodes but the tmp file is not deleted even after the build is completed.

          I observed that jenkins_file.zip is created in the node's workspace once the job starts building (as expected) but no file is copied/created in the node's java.io.tmpdir directory.

          The guide I used for configuring my job's parameters can be found here:
          https://wiki.jenkins-ci.org/display/JENKINS/Parameterized+Build

          orrc:
          I tried using your curl command with my job but it doesn't seem to set the parameters correctly. The parameters do not reflect on the http://jenkins.local:8080/job/my_job/build_number/parameters page. Although, I noticed this time, it automatically deleted the upload_*.tmp that was generated.

          Elgin Peteza added a comment - - edited Basically, I execute a job by POSTing to Jenkins using python's requests module via a remote PC. i.e. import requests with open( "file.zip" , "rb" ) as z: file_to_send = { "jenkins_file.zip" : z} requests.post( "http: //jenkins.local:8080/job/my_job/buildWithParameters" , files=file_to_send) An upload_*.tmp file is created in /tmp directory of the master after executing the script. The master then builds the job on one of its nodes but the tmp file is not deleted even after the build is completed. I observed that jenkins_file.zip is created in the node's workspace once the job starts building (as expected) but no file is copied/created in the node's java.io.tmpdir directory. The guide I used for configuring my job's parameters can be found here: https://wiki.jenkins-ci.org/display/JENKINS/Parameterized+Build orrc : I tried using your curl command with my job but it doesn't seem to set the parameters correctly. The parameters do not reflect on the http://jenkins.local:8080/job/my_job/build_number/parameters page. Although, I noticed this time, it automatically deleted the upload_*.tmp that was generated.

          Daniel Beck added a comment -

          orrc Jenkins itself (on job/foo/api) documents that parameterized builds should be started via /buildWithParameters. The POST to /build with JSON is just replicating what the browser does when you submit the form and IMO therefore more fragile/subject to change.

          Daniel Beck added a comment - orrc Jenkins itself (on job/foo/api) documents that parameterized builds should be started via /buildWithParameters. The POST to /build with JSON is just replicating what the browser does when you submit the form and IMO therefore more fragile/subject to change.

          Elgin Peteza added a comment - - edited

          Update:
          I encountered another problem with the upload_*.tmp files.

          I executed the job many times such that several builds will be queued on one node (and several upload_*.tmp files created in /tmp directory of the master), and then I restarted Jenkins in the middle. Those builds before restart were built successfully (upload_*.tmp files still not deleted). The remaining builds after restart, surprisingly, failed with the following error in the console:

          FATAL: null
          java.lang.NullPointerException
          	at hudson.model.FileParameterValue$2.setUp(FileParameterValue.java:143)
          	at hudson.model.Build$BuildExecution.doRun(Build.java:156)
          	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:537)
          	at hudson.model.Run.execute(Run.java:1744)
          	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
          	at hudson.model.ResourceController.execute(ResourceController.java:98)
          	at hudson.model.Executor.run(Executor.java:374)
          

          From previous successful builds, the console showed:

          Copying file to jenkins_file.zip
          

          All upload_*.tmp files were still in /tmp directory after the error occurred.

          Added:
          Maybe this is an entirely different bug. Based on my tests, only file parameters greater than 10kB generate upload_*.tmp files in /tmp directory. The error described above also occurred even for file parameters smaller than 10kB, which didn't create an upload_*.tmp in the first place.

          Elgin Peteza added a comment - - edited Update: I encountered another problem with the upload_*.tmp files. I executed the job many times such that several builds will be queued on one node (and several upload_*.tmp files created in /tmp directory of the master), and then I restarted Jenkins in the middle. Those builds before restart were built successfully ( upload_*.tmp files still not deleted). The remaining builds after restart, surprisingly, failed with the following error in the console: FATAL: null java.lang.NullPointerException at hudson.model.FileParameterValue$2.setUp(FileParameterValue.java:143) at hudson.model.Build$BuildExecution.doRun(Build.java:156) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:537) at hudson.model.Run.execute(Run.java:1744) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:98) at hudson.model.Executor.run(Executor.java:374) From previous successful builds, the console showed: Copying file to jenkins_file.zip All upload_*.tmp files were still in /tmp directory after the error occurred. Added: Maybe this is an entirely different bug. Based on my tests, only file parameters greater than 10kB generate upload_*.tmp files in /tmp directory. The error described above also occurred even for file parameters smaller than 10kB, which didn't create an upload_*.tmp in the first place.

          Brian Hinz added a comment -

          This is still an issue in 2.121.3, and there is no viable workaround that I can find.

           

          Brian Hinz added a comment - This is still an issue in 2.121.3, and there is no viable workaround that I can find.  

            bphinz Brian Hinz
            acwwat Anthony Wat
            Votes:
            2 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated: