Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-22088

Disk space leak with multiple copies of winstone-XXXX.jar in TEMP folder

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • core
    • Windows 8.1 x64, JDK 1.7.0_51, Jenkins 1.553

      Continued operation of Jenkins creates multiple copies of "winstone.jar" with a numeric suffix in TEMP folder (winstone12343543634634.jar). We had about 800Gb of these accumulated, slowing down the host system.

      We are currently running Jenkins directly (`java -jar jenkins.jar`) via Windows scheduled task.

          [JENKINS-22088] Disk space leak with multiple copies of winstone-XXXX.jar in TEMP folder

          Matt Hicks added a comment -

          This is a serious problem. I just discovered that my Jenkins installation, which I don't even use, has been filling my drive with these files for at least 2 months, taking up several hundred GB of space. This is on Mac OS X 10.9.

          Matt Hicks added a comment - This is a serious problem. I just discovered that my Jenkins installation, which I don't even use, has been filling my drive with these files for at least 2 months, taking up several hundred GB of space. This is on Mac OS X 10.9.

          Sean Dwyer added a comment -

          I had this issue also. In my case it was caused by the Jenkins service starting and failing constantly and creating a winstone*.jar file in the temp folder each time. This was happening because Jenkins was already running as a process when the service was trying to start. I fixed it by killing the java process for Jenkins in Task Manager, then starting the Jenkins Service.

          Sean Dwyer added a comment - I had this issue also. In my case it was caused by the Jenkins service starting and failing constantly and creating a winstone*.jar file in the temp folder each time. This was happening because Jenkins was already running as a process when the service was trying to start. I fixed it by killing the java process for Jenkins in Task Manager, then starting the Jenkins Service.

          Amanda Xu added a comment -

          Hello There. I am running into the same problem. Is there a workaround? My Jenkins version is 1.542. Thanks in advanced.

          Amanda Xu added a comment - Hello There. I am running into the same problem. Is there a workaround? My Jenkins version is 1.542. Thanks in advanced.

          PL Nech added a comment -

          Hi there,

          I have the same problem with Jenkins ver. 1.612 on Windows 7 64bits.

          PL Nech added a comment - Hi there, I have the same problem with Jenkins ver. 1.612 on Windows 7 64bits.

          Jay Beeckman added a comment -

          Same here on 1.627 on Windows 2008 R2 64 Bit. I'm going to manually clear it out, but this should be fixed.

          Jay Beeckman added a comment - Same here on 1.627 on Windows 2008 R2 64 Bit. I'm going to manually clear it out, but this should be fixed.

          This seems to happen on non-Windows as well; I have a Linux server and have a bunch in /tmp.

          Aaron D. Marasco added a comment - This seems to happen on non-Windows as well; I have a Linux server and have a bunch in /tmp .

          Deepak Koirala added a comment - - edited

          Is there a workaround yet? Do we need to manually delete all the winstoneXXXXXXXXXXXX.jar? Deleting these jars won't affect anything, right? We have Jenkins ver. 1.613, installed in windows server and facing the same issue. Almost 70 GB space is full within 3 months. And I saw most of the space is taken by winstone.jar in temp folder.

          Deepak Koirala added a comment - - edited Is there a workaround yet? Do we need to manually delete all the winstoneXXXXXXXXXXXX.jar? Deleting these jars won't affect anything, right? We have Jenkins ver. 1.613, installed in windows server and facing the same issue. Almost 70 GB space is full within 3 months. And I saw most of the space is taken by winstone.jar in temp folder.

          Same here on Windows 2008 Server.

          Noël Bardelot added a comment - Same here on Windows 2008 Server.

          Daniel added a comment -

          I just stumbled upon this Issue and used https://wiki.jenkins-ci.org/display/JENKINS/Tmp+Cleaner+Plugin as a workaround.
          Had to set all mentioned parameters with system properties when starting Jenkins but result was a cleaned /var/tmp after that.
          YMMV, but might be a valid workaround. HTH, Daniel

          Daniel added a comment - I just stumbled upon this Issue and used https://wiki.jenkins-ci.org/display/JENKINS/Tmp+Cleaner+Plugin as a workaround. Had to set all mentioned parameters with system properties when starting Jenkins but result was a cleaned /var/tmp after that. YMMV, but might be a valid workaround. HTH, Daniel

          Johan Smolinski added a comment - - edited

          I was also affected by this on Windows Server 2008, caused by an old jenkins process lying around while the service started another. A new winstone-XXXX.jar of 2 MB was created once every two seconds. I killed all java processes, deleted all files and restarted the service. Now there's only one of these jar files.

          I don't know the inners of jenkins, but there should be some sanity check before creating a new winstone-XXXX.jar.

          Johan Smolinski added a comment - - edited I was also affected by this on Windows Server 2008, caused by an old jenkins process lying around while the service started another. A new winstone-XXXX.jar of 2 MB was created once every two seconds. I killed all java processes, deleted all files and restarted the service. Now there's only one of these jar files. I don't know the inners of jenkins, but there should be some sanity check before creating a new winstone-XXXX.jar.

          Hello. Just run in this issue on our build server.
          Jenkins ver. 2.32.1.
          Windows Server 2008 R2 Standard 64 Bit.
          winstone-XXXX.jar was created and filled up disk space on C:. Solution was to kill all Jenkins/Java processes and restart Jenkins.

          Erwin Müller added a comment - Hello. Just run in this issue on our build server. Jenkins ver. 2.32.1. Windows Server 2008 R2 Standard 64 Bit. winstone-XXXX.jar was created and filled up disk space on C:. Solution was to kill all Jenkins/Java processes and restart Jenkins.

          Juri Strumpflohner added a comment - - edited

          same here. We just upgraded to 2.32.3 a couple of days ago and now I have this issue. Didn't have it before, at least I did not notice anything.

          The question is, can I delete them safely?

          Juri Strumpflohner added a comment - - edited same here. We just upgraded to 2.32.3 a couple of days ago and now I have this issue. Didn't have it before, at least I did not notice anything. The question is, can I delete them safely?

          juristr, you probably have an old process running from the previous version. Close all jenkins processes including any old ones, remove all of the temp files and start it up. That's how I solved it and I'm pretty sure it's safe.

          Johan Smolinski added a comment - juristr , you probably have an old process running from the previous version. Close all jenkins processes including any old ones, remove all of the temp files and start it up. That's how I solved it and I'm pretty sure it's safe.

          Joseph Qu added a comment -

          I can reproduce this error. Set jenkins as a windows service. then stop it. Install another application which is using 8080 port. restart the jenkins service. it will keep generating winstone.jar

          Joseph Qu added a comment - I can reproduce this error. Set jenkins as a windows service. then stop it. Install another application which is using 8080 port. restart the jenkins service. it will keep generating winstone.jar

          Peter Goins added a comment -

          We've seen this same issue running version 2.59.  In our case it is creating a new file each time we update Jenkins. Only the most recent file was locked by the running service - all other copies could be deleted.

          Peter Goins added a comment - We've seen this same issue running version 2.59.  In our case it is creating a new file each time we update Jenkins. Only the most recent file was locked by the running service - all other copies could be deleted.

          Our Jenkins instance on version 2.46.3 is also having this issue. It has been rapidly creating these files. For our instance, it seems to happen with more than just  restarting the service, as since April, 2017 we have gotten 35,000 copies of this file in our temp folder. 

          Taylor Williams added a comment - Our Jenkins instance on version 2.46.3 is also having this issue. It has been rapidly creating these files. For our instance, it seems to happen with more than just  restarting the service, as since April, 2017 we have gotten 35,000 copies of this file in our temp folder. 

          Mark R added a comment -

          We are experiencing this issue as well.

          Mark R added a comment - We are experiencing this issue as well.

          the jna-* folders are also not deleted

          Ivan Fernandez Calvo added a comment - the jna-* folders are also not deleted

          Jesse Glick added a comment -

          Please remember to add a link to your PR when you put one up for review. Otherwise it would be very hard to find the PR starting from JIRA.

          Jesse Glick added a comment - Please remember to add a link to your PR when you put one up for review. Otherwise it would be very hard to find the PR starting from JIRA.

          Jesse Glick added a comment -

          Since extras-executable-war already supports the --extractedFilesFolder option perhaps we could just clear the contents of this folder on startup if it was explicitly specified? Then using this option would be the solution, and we would just want to modify the native packages to specify it.

          Jesse Glick added a comment - Since extras-executable-war already supports the --extractedFilesFolder option perhaps we could just clear the contents of this folder on startup if it was explicitly specified? Then using this option would be the solution, and we would just want to modify the native packages to specify it.

          I added the check around the cleanup, now only do a cleanup if you set the --extractedFilesFolder=FOLDER property.

          Ivan Fernandez Calvo added a comment - I added the check around the cleanup, now only do a cleanup if you set the --extractedFilesFolder=FOLDER property.

          Jan Grieb added a comment -

          I experience this issue on a Win7 system which runs the Jenkins in the user space (not as service). 4 slaves are connected. Roughly every 5 minutes, a new winstone* file is created in the TEMP folder and never deleted.

          Jan Grieb added a comment - I experience this issue on a Win7 system which runs the Jenkins in the user space (not as service). 4 slaves are connected. Roughly every 5 minutes, a new winstone* file is created in the TEMP folder and never deleted.

          Code changed in jenkins
          User: Ivan Fernandez Calvo
          Path:
          src/main/java/Main.java
          http://jenkins-ci.org/commit/extras-executable-war/fe7acc205a251aa84dfb2bc50bccad1bc225426b
          Log:
          JENKINS-22088 Disk space leak with multiple copies of winstone-XXXX.jar in TEMP folder

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Ivan Fernandez Calvo Path: src/main/java/Main.java http://jenkins-ci.org/commit/extras-executable-war/fe7acc205a251aa84dfb2bc50bccad1bc225426b Log: JENKINS-22088 Disk space leak with multiple copies of winstone-XXXX.jar in TEMP folder

          Code changed in jenkins
          User: Ivan Fernandez Calvo
          Path:
          http://jenkins-ci.org/commit/extras-executable-war/ebb0c7fa2998c5cb450eeabc40621424beb1ebf7
          Log:
          Merge branch 'JENKINS-22088' of github.com:kuisathaverat/extras-executable-war into JENKINS-22088

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Ivan Fernandez Calvo Path: http://jenkins-ci.org/commit/extras-executable-war/ebb0c7fa2998c5cb450eeabc40621424beb1ebf7 Log: Merge branch ' JENKINS-22088 ' of github.com:kuisathaverat/extras-executable-war into JENKINS-22088

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          src/main/java/Main.java
          http://jenkins-ci.org/commit/extras-executable-war/3aa664c3880534e98aac9dd496cf67afeeef1f16
          Log:
          Merge pull request #12 from kuisathaverat/JENKINS-22088

          JENKINS-22088 Disk space leak

          Compare: https://github.com/jenkinsci/extras-executable-war/compare/a8f310ff07f6...3aa664c38805

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: src/main/java/Main.java http://jenkins-ci.org/commit/extras-executable-war/3aa664c3880534e98aac9dd496cf67afeeef1f16 Log: Merge pull request #12 from kuisathaverat/ JENKINS-22088 JENKINS-22088 Disk space leak Compare: https://github.com/jenkinsci/extras-executable-war/compare/a8f310ff07f6...3aa664c38805

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          war/pom.xml
          http://jenkins-ci.org/commit/jenkins/54c5872d1b27d244b250d518d6846351be13f503
          Log:
          JENKINS-22088 - Upgrade Extras Executable War from 1.36 to 1.37

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: war/pom.xml http://jenkins-ci.org/commit/jenkins/54c5872d1b27d244b250d518d6846351be13f503 Log: JENKINS-22088 - Upgrade Extras Executable War from 1.36 to 1.37

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          war/pom.xml
          http://jenkins-ci.org/commit/jenkins/bff2d2a399018655e3b54e1883f5efff176f3f8c
          Log:
          Merge pull request #3223 from oleg-nenashev/extras-executable-war/1.37

          JENKINS-22088 - Upgrade Extras Executable War from 1.36 to 1.37

          Compare: https://github.com/jenkinsci/jenkins/compare/2a6fc653ee7b...bff2d2a39901

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: war/pom.xml http://jenkins-ci.org/commit/jenkins/bff2d2a399018655e3b54e1883f5efff176f3f8c Log: Merge pull request #3223 from oleg-nenashev/extras-executable-war/1.37 JENKINS-22088 - Upgrade Extras Executable War from 1.36 to 1.37 Compare: https://github.com/jenkinsci/jenkins/compare/2a6fc653ee7b...bff2d2a39901

          Peter Hannon added a comment - - edited

          I am still seeing this issue with Jenkins v 2.121.3. From jenkins release notes: https://jenkins.io/changelog-stable/#v2.107.1 I thought this had been fixed in v2.107.1

          I noticed recently that there were several winstone jars in my /tmp. I manually removed all but the most recent.  I then did a safeRestart of Jenkins and there were then 2 winstone jars in the /tmp directory of the master Jenkins node.

          ls -lrt winstone*.jar
          -rw-r--r-- 1 jenkins users 2146767 2019-03-04 12:23 winstone9180695101677339973.jar
          -rw-r--r-- 1 jenkins users 2146767 2019-04-04 11:21 winstone8639977864837551161.jar
          

          We are running the Jenkins war directly via a shell script

          #!/bin/bash
          nohup /home/jenkins/jdk8/bin/java -jar /home/jenkins/jenkins.war --httpPort=8081 > /home/jenkins/jenkins.log 2>&1 &
          

           

          I'm not sure if it's related but I am also seeing some empty jetty directories with similar timestamps:

          ls -ldrt jetty*
          drwxr-xr-x 2 jenkins users 4096 2019-03-04 12:23 jetty-0.0.0.0-8081-war-_-any-2428298004054280574.dir
          drwxr-xr-x 2 jenkins users 4096 2019-04-04 11:21 jetty-0.0.0.0-8081-war-_-any-7220745071344149958.dir
          

           

           

          Peter Hannon added a comment - - edited I am still seeing this issue with Jenkins v 2.121.3. From jenkins release notes:  https://jenkins.io/changelog-stable/#v2.107.1  I thought this had been fixed in v2.107.1 I noticed recently that there were several winstone jars in my /tmp. I manually removed all but the most recent.  I then did a safeRestart of Jenkins and there were then 2 winstone jars in the /tmp directory of the master Jenkins node. ls -lrt winstone*.jar -rw-r--r-- 1 jenkins users 2146767 2019-03-04 12:23 winstone9180695101677339973.jar -rw-r--r-- 1 jenkins users 2146767 2019-04-04 11:21 winstone8639977864837551161.jar We are running the Jenkins war directly via a shell script #!/bin/bash nohup /home/jenkins/jdk8/bin/java -jar /home/jenkins/jenkins.war --httpPort=8081 > /home/jenkins/jenkins.log 2>&1 &   I'm not sure if it's related but I am also seeing some empty jetty directories with similar timestamps: ls -ldrt jetty* drwxr-xr-x 2 jenkins users 4096 2019-03-04 12:23 jetty-0.0.0.0-8081-war-_-any-2428298004054280574.dir drwxr-xr-x 2 jenkins users 4096 2019-04-04 11:21 jetty-0.0.0.0-8081-war-_-any-7220745071344149958.dir    

          Peter Hannon added a comment -

          ifernandezcalvo thanks for the reply, if you see my comment (the shell script), I'm running this in linux, so the windows service change is not applicable for me.

          Are you saying for a linux environment, this generated winstone jar file needs to be cleaned manually?

          Peter Hannon added a comment - ifernandezcalvo thanks for the reply, if you see my comment (the shell script), I'm running this in linux, so the windows service change is not applicable for me. Are you saying for a linux environment, this generated winstone jar file needs to be cleaned manually?

          ptha sorry, I've replied something unrelated.

          You have to pass an argument to enable the cleanup `--extractedFilesFolder=FOLDER_PATH`

          https://github.com/kuisathaverat/extras-executable-war/blob/8f00f05c22cb4953ed0a034e0d7ef35c0076f99f/src/main/java/Main.java#L229-L232

          Ivan Fernandez Calvo added a comment - ptha sorry, I've replied something unrelated. You have to pass an argument to enable the cleanup `--extractedFilesFolder=FOLDER_PATH` https://github.com/kuisathaverat/extras-executable-war/blob/8f00f05c22cb4953ed0a034e0d7ef35c0076f99f/src/main/java/Main.java#L229-L232

          Peter Hannon added a comment -

          ifernandezcalvo thanks again!

          As per your suggestion, I updated my start shell script to:

           

          #!/bin/bash
          nohup /home/jenkins/jdk8/bin/java -jar /home/jenkins/jenkins.war --httpPort=8081 --extractedFilesFolder=/tmp > /home/jenkins/jenkins.log 2>&1 &
          

          Now the extra winstone jars are cleaned up on startup, I also tested a safeRestart and again there was only 1 winstone jar file (for the running Jenkins).

          So it's solved for me, cheers!

           

          Peter Hannon added a comment - ifernandezcalvo  thanks again! As per your suggestion, I updated my start shell script to:   #!/bin/bash nohup /home/jenkins/jdk8/bin/java -jar /home/jenkins/jenkins.war --httpPort=8081 --extractedFilesFolder=/tmp > /home/jenkins/jenkins.log 2>&1 & Now the extra winstone jars are cleaned up on startup, I also tested a safeRestart and again there was only 1 winstone jar file (for the running Jenkins). So it's solved for me, cheers!  

          This issue is happening with my current Jenkins build 2.156 on windows. Is there any solution for windows for this. 

          Disk is getting full, due to so many winstone*.jar in C:\...\AppData\Local\Temp folder

          Shashank Kulshrestha added a comment - This issue is happening with my current Jenkins build 2.156 on windows. Is there any solution for windows for this.  Disk is getting full, due to so many winstone*.jar in C:\...\AppData\Local\Temp folder

          shashankkulsh Did you read the comment before yours? there you have the solution *--extractedFilesFolder=c:\TMP-FOLDER-PATH*

          Ivan Fernandez Calvo added a comment - shashankkulsh Did you read the comment before yours? there you have the solution * --extractedFilesFolder=c:\TMP-FOLDER-PATH *

          Matt Barna added a comment -

          Is there any plan to fix this? Seems wonky that Jenkins knows where to write the jar, but not where to go to delete it.

          Matt Barna added a comment - Is there any plan to fix this? Seems wonky that Jenkins knows where to write the jar, but not where to go to delete it.

          I just had my Windows 2012R2 Jenkins 2.257 server fall to its knees for the second time in a week after something threw a spanner at Jenkins (started as a service). Jenkins was convinced that it needed to restart itself, which failed due to address already in use, so try again and again and again, dumping a new copy of winstone#.jar each time until out of disk space.

          This bizarre behavior seems to be the result of how the Jenkins service is created by default, first failure: restart, second failure: restart, subsequent failures: restart.

          For now I've set subsequent failures to 'take no action' - hopefully this will ease the pain.

          Stefan Drissen added a comment - I just had my Windows 2012R2 Jenkins 2.257 server fall to its knees for the second time in a week after something threw a spanner at Jenkins (started as a service). Jenkins was convinced that it needed to restart itself, which failed due to address already in use, so try again and again and again, dumping a new copy of winstone#.jar each time until out of disk space. This bizarre behavior seems to be the result of how the Jenkins service is created by default, first failure: restart, second failure: restart, subsequent failures: restart. For now I've set subsequent failures to 'take no action' - hopefully this will ease the pain.

          I am currently experiencing this issue with a jenkins 2.401.1 (LTS) running on Windows server 2016. Still seems to be an active issue...

          Christoph Fetzer added a comment - I am currently experiencing this issue with a jenkins 2.401.1 (LTS) running on Windows server 2016. Still seems to be an active issue...

          The issue was fixed and nobody report a hit in five years, Could you open a new one with the details of your environment and reference this one?

          Ivan Fernandez Calvo added a comment - The issue was fixed and nobody report a hit in five years, Could you open a new one with the details of your environment and reference this one?

            ifernandezcalvo Ivan Fernandez Calvo
            t0yv0 Anton Tayanovskyy
            Votes:
            20 Vote for this issue
            Watchers:
            34 Start watching this issue

              Created:
              Updated:
              Resolved: