Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-12699

Temp upload files from file param not removed after transferred to slave

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: In Review (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: core
    • Labels:
    • Environment:
      OS: Linux ara02 2.6.9-103.ELsmp #1 SMP Fri Nov 11 14:34:02 EST 2011 i686 i686 i386 GNU/Linux
      JRE: java version "1.6.0_16"
      Jenkins: 1.442
    • Similar Issues:

      Description

      I currently have a master-slave configuration on Jenkins 1.442 running on a Linux box. Lately I started using file parameter for one of my jobs, and noticed that disk space was decreasing rather quickly on the master system. Upon examination it turns out that file parameter is creating files under /tmp, with a naming convention upload_*.tmp, when the file parameters are uploaded. I've done some experiment and noticed that:

      • If the job is running on master, the tmp file is moved into the workspace correctly.
      • If the job is running on slave, the tmp file is uploaded to the slave workspace correctly, but the upload_*.tmp file remains in /tmp.

      I am using a file parameter for a 100 MB file, so it is very easy to collect GBs worth of temporary files, until my master runs out of disk space. For now I have added a cron job to delete these upload_*.tmp files, but it's not safe if the script runs at the same time as a user uploading a file param for a job.

      Let me know what other debugging information I need to provide, if any. Please investigate, thanks.

        Attachments

          Issue Links

            Activity

            Hide
            evernat evernat added a comment -

            Is it reproduced with a recent Jenkins version?

            Show
            evernat evernat added a comment - Is it reproduced with a recent Jenkins version?
            Hide
            evernat evernat added a comment -

            No response from the reporter, so resolving as incomplete.

            Show
            evernat evernat added a comment - No response from the reporter, so resolving as incomplete.
            Hide
            orrc Christopher Orr added a comment -

            Reopening as per JENKINS-29121.

            Show
            orrc Christopher Orr added a comment - Reopening as per JENKINS-29121 .
            Hide
            orrc Christopher Orr added a comment -

            I'm not seeing this behaviour with large file parameters, whether the build is executed on master or a slave.

            But I'm not using buildWithParameters, but rather the build format documented here:
            https://wiki.jenkins-ci.org/display/JENKINS/Remote+access+API#RemoteaccessAPI-Submittingjobs

            i.e.

            curl -v -X POST http://jenkins.local:8080/job/upload-test/build \
              --form f0=@/data/fw123.bin \
              --form json='{"parameter": [{"name":"firmware.bin", "file":"f0"}]}'
            

            When the job is running, I see "Copying file to firmware.bin" in the console output.
            But checking /tmp and the java.io.tmpdir directory on the slave while this is happening shows no temporary files being created.

            Elgin Peteza: Can you provide more detail on exactly how you're starting a job?

            Show
            orrc Christopher Orr added a comment - I'm not seeing this behaviour with large file parameters, whether the build is executed on master or a slave. But I'm not using buildWithParameters , but rather the build format documented here: https://wiki.jenkins-ci.org/display/JENKINS/Remote+access+API#RemoteaccessAPI-Submittingjobs i.e. curl -v -X POST http://jenkins.local:8080/job/upload-test/build \ --form f0=@/data/fw123.bin \ --form json='{"parameter": [{"name":"firmware.bin", "file":"f0"}]}' When the job is running, I see "Copying file to firmware.bin" in the console output. But checking /tmp and the java.io.tmpdir directory on the slave while this is happening shows no temporary files being created. Elgin Peteza : Can you provide more detail on exactly how you're starting a job?
            Hide
            toshee2000 Elgin Peteza added a comment - - edited

            Basically, I execute a job by POSTing to Jenkins using python's requests module via a remote PC.
            i.e.

            import requests
            with open("file.zip", "rb") as z:
                file_to_send = {"jenkins_file.zip": z}
                requests.post("http://jenkins.local:8080/job/my_job/buildWithParameters", files=file_to_send)
            

            An upload_*.tmp file is created in /tmp directory of the master after executing the script. The master then builds the job on one of its nodes but the tmp file is not deleted even after the build is completed.

            I observed that jenkins_file.zip is created in the node's workspace once the job starts building (as expected) but no file is copied/created in the node's java.io.tmpdir directory.

            The guide I used for configuring my job's parameters can be found here:
            https://wiki.jenkins-ci.org/display/JENKINS/Parameterized+Build

            Christopher Orr:
            I tried using your curl command with my job but it doesn't seem to set the parameters correctly. The parameters do not reflect on the http://jenkins.local:8080/job/my_job/build_number/parameters page. Although, I noticed this time, it automatically deleted the upload_*.tmp that was generated.

            Show
            toshee2000 Elgin Peteza added a comment - - edited Basically, I execute a job by POSTing to Jenkins using python's requests module via a remote PC. i.e. import requests with open( "file.zip" , "rb" ) as z: file_to_send = { "jenkins_file.zip" : z} requests.post( "http: //jenkins.local:8080/job/my_job/buildWithParameters" , files=file_to_send) An upload_*.tmp file is created in /tmp directory of the master after executing the script. The master then builds the job on one of its nodes but the tmp file is not deleted even after the build is completed. I observed that jenkins_file.zip is created in the node's workspace once the job starts building (as expected) but no file is copied/created in the node's java.io.tmpdir directory. The guide I used for configuring my job's parameters can be found here: https://wiki.jenkins-ci.org/display/JENKINS/Parameterized+Build Christopher Orr : I tried using your curl command with my job but it doesn't seem to set the parameters correctly. The parameters do not reflect on the http://jenkins.local:8080/job/my_job/build_number/parameters page. Although, I noticed this time, it automatically deleted the upload_*.tmp that was generated.
            Hide
            danielbeck Daniel Beck added a comment -

            Christopher Orr Jenkins itself (on job/foo/api) documents that parameterized builds should be started via /buildWithParameters. The POST to /build with JSON is just replicating what the browser does when you submit the form and IMO therefore more fragile/subject to change.

            Show
            danielbeck Daniel Beck added a comment - Christopher Orr Jenkins itself (on job/foo/api) documents that parameterized builds should be started via /buildWithParameters. The POST to /build with JSON is just replicating what the browser does when you submit the form and IMO therefore more fragile/subject to change.
            Hide
            toshee2000 Elgin Peteza added a comment - - edited

            Update:
            I encountered another problem with the upload_*.tmp files.

            I executed the job many times such that several builds will be queued on one node (and several upload_*.tmp files created in /tmp directory of the master), and then I restarted Jenkins in the middle. Those builds before restart were built successfully (upload_*.tmp files still not deleted). The remaining builds after restart, surprisingly, failed with the following error in the console:

            FATAL: null
            java.lang.NullPointerException
            	at hudson.model.FileParameterValue$2.setUp(FileParameterValue.java:143)
            	at hudson.model.Build$BuildExecution.doRun(Build.java:156)
            	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:537)
            	at hudson.model.Run.execute(Run.java:1744)
            	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
            	at hudson.model.ResourceController.execute(ResourceController.java:98)
            	at hudson.model.Executor.run(Executor.java:374)
            

            From previous successful builds, the console showed:

            Copying file to jenkins_file.zip
            

            All upload_*.tmp files were still in /tmp directory after the error occurred.

            Added:
            Maybe this is an entirely different bug. Based on my tests, only file parameters greater than 10kB generate upload_*.tmp files in /tmp directory. The error described above also occurred even for file parameters smaller than 10kB, which didn't create an upload_*.tmp in the first place.

            Show
            toshee2000 Elgin Peteza added a comment - - edited Update: I encountered another problem with the upload_*.tmp files. I executed the job many times such that several builds will be queued on one node (and several upload_*.tmp files created in /tmp directory of the master), and then I restarted Jenkins in the middle. Those builds before restart were built successfully ( upload_*.tmp files still not deleted). The remaining builds after restart, surprisingly, failed with the following error in the console: FATAL: null java.lang.NullPointerException at hudson.model.FileParameterValue$2.setUp(FileParameterValue.java:143) at hudson.model.Build$BuildExecution.doRun(Build.java:156) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:537) at hudson.model.Run.execute(Run.java:1744) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:98) at hudson.model.Executor.run(Executor.java:374) From previous successful builds, the console showed: Copying file to jenkins_file.zip All upload_*.tmp files were still in /tmp directory after the error occurred. Added: Maybe this is an entirely different bug. Based on my tests, only file parameters greater than 10kB generate upload_*.tmp files in /tmp directory. The error described above also occurred even for file parameters smaller than 10kB, which didn't create an upload_*.tmp in the first place.
            Hide
            bphinz Brian Hinz added a comment -

            This is still an issue in 2.121.3, and there is no viable workaround that I can find.

             

            Show
            bphinz Brian Hinz added a comment - This is still an issue in 2.121.3, and there is no viable workaround that I can find.  

              People

              Assignee:
              bphinz Brian Hinz
              Reporter:
              acwwat Anthony Wat
              Votes:
              2 Vote for this issue
              Watchers:
              8 Start watching this issue

                Dates

                Created:
                Updated: