• • • Issue 4249 has been marked as a duplicate of this issue. ***

I've noticed another case where the temp folder for the last test to run does
not get removed. This appears to be because TestEnvironment.dispose() calls
TemporaryDirectoryAllocator.disposeAsync() and the tests exit before this thread
completes (even though it is not a daemon thread). I see this on OSX.

Problem still exists.

In my environment, the output becomes like this:

...
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 30.876 sec
Failed to load native POSIX impl; falling back on Java impl. Unsupported OS.
java.io.IOException: Unable to delete C:\Users\user\AppData\Local\Temp\hudson4413809435965306193tmp\credentials\WEB-INF\lib\findbugs-annotations-1.3.9-1.jar
	at hudson.Util.deleteFile(Util.java:256)
	at hudson.Util.deleteRecursive(Util.java:308)
	at hudson.Util.deleteContentsRecursive(Util.java:205)
	at hudson.Util.deleteRecursive(Util.java:299)
	at hudson.Util.deleteContentsRecursive(Util.java:205)
	at hudson.Util.deleteRecursive(Util.java:299)
	at hudson.Util.deleteContentsRecursive(Util.java:205)
	at hudson.Util.deleteRecursive(Util.java:299)
	at hudson.Util.deleteContentsRecursive(Util.java:205)
	at hudson.Util.deleteRecursive(Util.java:299)
	at org.jvnet.hudson.test.TestPluginManager$1.run(TestPluginManager.java:130)

Results :

Tests run: 7, Failures: 0, Errors: 0, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------

• The test succeeds, but fails to delete the temporary folder.
• TestPluginManager seems to expand common plugins to the temporary folder used by Jenkins instance for the test.
• For JVM holds handles of jar files, the shutdown hook registered by TestPluginManager fails to delete the temporary directory.

I verified that following code resolves the problem:

// when your test classes run with JenkinsRule:
    @Rule
    public JenkinsRule j = new JenkinsRule(){
        protected void before() throws Throwable {
            setPluginManager(null);
            super.before();
        }
    };

// when your test classes derived from HudsonTestCase:
    @Override
    protected void setUp() throws Exception
    {
        setPluginManager(null);
        super.setUp();
    }

• This takes effects only when running with -Dtest=(test class)
• When run as "mvn test" (without -Dtest=XXXX), InjectTests runs and TestPluginManager is used.

Stop InjectedTests with following in pom.xml:

  <build>
    <plugins>
      <plugin>
        <groupId>org.jenkins-ci.tools</groupId>
        <artifactId>maven-hpi-plugin</artifactId>
        <extensions>true</extensions>
        <configuration>
          <disabledTestInjection>true</disabledTestInjection>
        </configuration>
      </plugin>
    </plugins>
  </build>

• ${java.io.tmp}/hudsonXXXXtest
  • Not deleted with JenkinsRule in Jenkins < 1.482 (5dd905f2a2)
• ${java.io.tmp}/hudsonXXXXtmp
  • Not deleted with Jenkins < 1.510 (a4d4305124)
  • Fails to be deleted with Jenkins >= 1.510

I think that TestPluginManager doesn't need to use different directories every time.
Using a fixed directory resolves the increase of temporary files.

Sent a pull request to make TestPluginManager use a fixed directory.
https://github.com/jenkinsci/jenkins/pull/780

A comment by jglick in the pull request.

Using a fixed directory seems dangerous—it means that different parallel test runs could clobber one another, or tests might pick up obsolete plugin versions from an older test run, etc.

I would rather try to diagnose and fix the failure to delete the JAR files. Typically this just means that someone is forgetting to close a JarFile, which can be solved by calling PluginWrapper.releaseClassLoader.

This is smarter way, and I'll try this.
Abondoned the pull request.

jglick is right.
I found that just removing TestPluginManager#stop resolves this problem.

But I'm not sure it is safe to remove TestPluginManager#stop.
This results in releasing classes after each tests, and may result:

• Built-in plugins does not work properly after the first test finishes.
• Plugins are reloaded each tests, and tests gets slow.

I have to test the behavior.

Integrated in jenkins_main_trunk #2630
[FIX JENKINS-4409] Release plugins and class loaders when JVM shuts down. (Revision 2b9a083fbf2b9f4946ffa01e35dffea2b20a98d1)

Result = SUCCESS
kohsuke : 2b9a083fbf2b9f4946ffa01e35dffea2b20a98d1
Files :

• test/src/main/java/org/jvnet/hudson/test/TestPluginManager.java

Code changed in jenkins
User: ikedam
Path:
test/src/main/java/org/jvnet/hudson/test/TestPluginManager.java
http://jenkins-ci.org/commit/jenkins/2b9a083fbf2b9f4946ffa01e35dffea2b20a98d1
Log:
[FIX JENKINS-4409] Release plugins and class loaders when JVM shuts down.

It should be fixed in 1.520, in https://github.com/jenkinsci/jenkins/pull/791, but still happens.

java.io.IOException: Unable to delete C:\Users\ikedam\AppData\Local\Temp\hudson1720733180692388407tmp\credentials\WEB-INF\lib\classes.jar
        at hudson.Util.deleteFile(Util.java:253)
        at hudson.Util.deleteRecursive(Util.java:305)
        at hudson.Util.deleteContentsRecursive(Util.java:202)
        at hudson.Util.deleteRecursive(Util.java:296)
        at hudson.Util.deleteContentsRecursive(Util.java:202)
        at hudson.Util.deleteRecursive(Util.java:296)
        at hudson.Util.deleteContentsRecursive(Util.java:202)
        at hudson.Util.deleteRecursive(Util.java:296)
        at hudson.Util.deleteContentsRecursive(Util.java:202)
        at hudson.Util.deleteRecursive(Util.java:296)
        at org.jvnet.hudson.test.TestPluginManager$1.run(TestPluginManager.java:146)

Hmm....

https://github.com/jenkinsci/jenkins/pull/791 works for 1.518, but not for 1.519.
Some changes in 1.519 may affect.

Hmm... seems affected by f7330d7.
classes.jar seems not unloaded.

This is caused by com/cloudbees/plugins/credentials/Messages.properties .
Excluding that file from classes.jar and expanding that to the file system fixes the problem.

Excluding *.properties and META-INF/annotations/hudson.Extension from classes.jar avoids the problem:

But excluding hudson.Extension may have affected the behavior of loading classes.

This may be an issue for JVM...
I could reproduce the problem that jar file cannot be deleted:

    public static void main(String[] args) throws Exception
    {
        File libFile = new File("lib/classes.jar");
        if(!libFile.exists())
        {
            System.out.println(String.format("Not exist: %s", libFile.getPath()));
            System.exit(1);
        }
        
        {
            URLClassLoader loader = new URLClassLoader(new URL[]{libFile.toURI().toURL()});
            
            Enumeration<URL> urls = loader.getResources("META-INF/annotations/hudson.Extension");
            while(urls.hasMoreElements())
            {
                System.out.println(urls.nextElement());
            }
        }
        System.gc();
        System.gc();
        System.gc();
        
        if(!libFile.delete())
        {
            System.out.println(String.format("Failed to delete: %s", libFile.getPath()));
            System.exit(1);
        }
    }

In Java 1.7, this could be resolved with:

        {
            URLClassLoader loader = new URLClassLoader(new URL[]{libFile.toURI().toURL()});
            
            Enumeration<URL> urls = loader.getResources("META-INF/annotations/hudson.Extension");
            while(urls.hasMoreElements())
            {
                System.out.println(urls.nextElement());
            }
+           ((java.io.Closeable)loader).close();
        }

And this is what hudson.PluginWrapper#releaseClassLoader does.

Hi there. I'm not sure where things are with this.

I am working on a plugin (windows for development , linux for CI). Of course,the linux part works fine

I find that consistently, when setting my POM to 1.519 it does a maven "Install" no problems.

I find that as soon as I go anywhere higher than 1.519, windows gets the same err... I have tried a whole bunch of version up to 1.533

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.008 sec
Oct 01, 2013 1:03:59 AM hudson.PluginWrapper stop
INFO: Stopping ironmq-notifier
java.nio.file.FileSystemException: C:\Users\mike\AppData\Local\Temp\hudson6898587435897076938tmp\credentials\WEB-INF\lib\classes.jar: The process cannot access the file because it is being used by another process.

-I've done a bit of research and found that it could somehow be related to Jetty... Is this possible?

This is the document I'm talking about. http://docs.codehaus.org/display/JETTY/Files+locked+on+Windows

I haven't looked that far in the code, but did the "credentials" logic change between 1.519 and the next version?

I am not sure if I helped here. I hope I did.

classes.jar is generated since Jenkins 1.519.
Why the error happens with credentials plugin seems that credentials plugin comes to the first ordered alphabetically. If it did not happen with credentials plugin, it would just happen with the next plugin.

I have just submitted a couple of PR that fixes this issue on my Windows machine.

• https://github.com/jglick/sezpoz/pull/9
• https://github.com/kohsuke/localizer/pull/9

Essentially, the issue is caused by URLClassLoader that is caching file handles for later reuse (cf. https://bugs.openjdk.java.net/browse/JDK-8013099).

I have no idea yet about the possible performance issues it may trigger, but definitely these were the causes for file handle retention.
Kudos to kohsuke for his file leak detector, that really helped to pinpoint the leak location.

Fantastic! This problem have bothered me for long time.
Can calling URLConnection.setDefaultUseCaches(false) be a workaround? (not tested yet)

I verified URLConnection.setDefaultUseCaches(false) works as expected.

I wrote workarounds on
https://wiki.jenkins-ci.org/display/JENKINS/Unit+Test+on+Windows#UnitTestonWindows-Jenkinscorecachesandkeepfilehandles

We'd better to consider not to modify sezpoz and localizer but to modify test classes in Jenkins.

Code changed in jenkins
User: ikedam
Path:
test/src/main/java/org/jvnet/hudson/test/HudsonTestCase.java
test/src/main/java/org/jvnet/hudson/test/JenkinsRule.java
http://jenkins-ci.org/commit/jenkins/c4e5fbdbc9c644f1afcae031b3b4638ad5297238
Log:
[FIXED JENKINS-4409] Disable URLConnection.useCache in tests on Windows. It prevents deleting temporary directories when tests finish.

Code changed in jenkins
User: Jesse Glick
Path:
test/src/main/java/org/jvnet/hudson/test/HudsonTestCase.java
test/src/main/java/org/jvnet/hudson/test/JenkinsRule.java
http://jenkins-ci.org/commit/jenkins/c8abbfc9728aa7a1bff415e6f4e624209ceb0020
Log:
Merge pull request #1572 from ikedam/feature/JENKINS-4409_DisableURLConnectionCache

JENKINS-4409 Disable URLConnection.useCache in tests on Windows

Compare: https://github.com/jenkinsci/jenkins/compare/f918dd84acde...c8abbfc9728a

Integrated in jenkins_main_trunk #3980
[FIXED JENKINS-4409] Disable URLConnection.useCache in tests on Windows. It prevents deleting temporary directories when tests finish. (Revision c4e5fbdbc9c644f1afcae031b3b4638ad5297238)

Result = SUCCESS
devld : c4e5fbdbc9c644f1afcae031b3b4638ad5297238
Files :

• test/src/main/java/org/jvnet/hudson/test/JenkinsRule.java
• test/src/main/java/org/jvnet/hudson/test/HudsonTestCase.java

Code changed in jenkins
User: Kohsuke Kawaguchi
Path:
cli/pom.xml
core/pom.xml
licenseCompleter.groovy
pom.xml
http://jenkins-ci.org/commit/jenkins/b62c927a27d5928b8577c298b06f7d819cb18ad2
Log:
[FIXED JENKINS-4409] Integrated a newer version of localizer

Integrated in jenkins_main_trunk #4017
[FIXED JENKINS-4409] Integrated a newer version of localizer (Revision b62c927a27d5928b8577c298b06f7d819cb18ad2)

Result = SUCCESS
kohsuke : b62c927a27d5928b8577c298b06f7d819cb18ad2
Files :

• cli/pom.xml
• pom.xml
• core/pom.xml
• licenseCompleter.groovy

Backported against 1.596.3 without [FIXED JENKINS-4409] Integrated a newer version of localizer that is a) not required and b) not soaked yet.

Code changed in jenkins
User: ikedam
Path:
test/src/main/java/org/jvnet/hudson/test/HudsonTestCase.java
test/src/main/java/org/jvnet/hudson/test/JenkinsRule.java
http://jenkins-ci.org/commit/jenkins/9bce20951c3ffe8e83a18a5a8d5743e183ebf3a8
Log:
[FIXED JENKINS-4409] Disable URLConnection.useCache in tests on Windows. It prevents deleting temporary directories when tests finish.

(cherry picked from commit c4e5fbdbc9c644f1afcae031b3b4638ad5297238)

Code changed in jenkins
User: Daniel Spilker
Path:
job-dsl-plugin/build.gradle
http://jenkins-ci.org/commit/job-dsl-plugin/bee907f1b74fee7718d010d352cc051308317893
Log:
added workaround for JENKINS-4409 to avoid leaking files into the system temp directory

Integrated in jenkins_main_trunk #4292
[FIXED JENKINS-4409] Disable URLConnection.useCache in tests on Windows. It prevents deleting temporary directories when tests finish. (Revision 9bce20951c3ffe8e83a18a5a8d5743e183ebf3a8)

Result = UNSTABLE
ogondza : 9bce20951c3ffe8e83a18a5a8d5743e183ebf3a8
Files :

• test/src/main/java/org/jvnet/hudson/test/HudsonTestCase.java
• test/src/main/java/org/jvnet/hudson/test/JenkinsRule.java

Code changed in jenkins
User: ikedam
Path:
test/src/main/java/org/jvnet/hudson/test/TestPluginManager.java
http://jenkins-ci.org/commit/jenkins-test-harness/bfb70904a4c3cbeb90f1e55766ea2456695b73ea
Log:
[FIX JENKINS-4409] Release plugins and class loaders when JVM shuts down.

Originally-Committed-As: 2b9a083fbf2b9f4946ffa01e35dffea2b20a98d1

Code changed in jenkins
User: ikedam
Path:
test/src/main/java/org/jvnet/hudson/test/HudsonTestCase.java
test/src/main/java/org/jvnet/hudson/test/JenkinsRule.java
http://jenkins-ci.org/commit/jenkins-test-harness/cb5bb384b1b995c08bdb2843d9a1ad2b1166066b
Log:
[FIXED JENKINS-4409] Disable URLConnection.useCache in tests on Windows. It prevents deleting temporary directories when tests finish.

Originally-Committed-As: c4e5fbdbc9c644f1afcae031b3b4638ad5297238

Code changed in jenkins
User: Jesse Glick
Path:
test/src/main/java/org/jvnet/hudson/test/HudsonTestCase.java
test/src/main/java/org/jvnet/hudson/test/JenkinsRule.java
http://jenkins-ci.org/commit/jenkins-test-harness/bd3655c132a175255598d6c13cd7d8b231e5c1a3
Log:
Merge pull request #1572 from ikedam/feature/JENKINS-4409_DisableURLConnectionCache

JENKINS-4409 Disable URLConnection.useCache in tests on Windows
Originally-Committed-As: c8abbfc9728aa7a1bff415e6f4e624209ceb0020

Code changed in jenkins
User: ikedam
Path:
test/src/main/java/org/jvnet/hudson/test/HudsonTestCase.java
test/src/main/java/org/jvnet/hudson/test/JenkinsRule.java
http://jenkins-ci.org/commit/jenkins-test-harness/3b839f41c16d0c5bc7d96a238c7beaac87885f96
Log:
[FIXED JENKINS-4409] Disable URLConnection.useCache in tests on Windows. It prevents deleting temporary directories when tests finish.

(cherry picked from commit c4e5fbdbc9c644f1afcae031b3b4638ad5297238)

Originally-Committed-As: 9bce20951c3ffe8e83a18a5a8d5743e183ebf3a8

Code changed in jenkins
User: Christian Pönisch
Path:
src/test/java/de/tracetronic/jenkins/plugins/ecutest/SystemTestBase.java
src/test/java/de/tracetronic/jenkins/plugins/ecutest/TestPluginManagerCleanUp.java
src/test/java/de/tracetronic/jenkins/plugins/ecutest/report/atx/ATXPublisherST.java
src/test/java/de/tracetronic/jenkins/plugins/ecutest/report/atx/ATXPublisherTest.java
src/test/java/de/tracetronic/jenkins/plugins/ecutest/report/generator/ReportGeneratorPublisherST.java
src/test/java/de/tracetronic/jenkins/plugins/ecutest/report/generator/ReportGeneratorPublisherTest.java
src/test/java/de/tracetronic/jenkins/plugins/ecutest/report/junit/JUnitPublisherST.java
src/test/java/de/tracetronic/jenkins/plugins/ecutest/report/junit/JUnitPublisherTest.java
src/test/java/de/tracetronic/jenkins/plugins/ecutest/report/log/ETLogPublisherST.java
src/test/java/de/tracetronic/jenkins/plugins/ecutest/report/log/ETLogPublisherTest.java
src/test/java/de/tracetronic/jenkins/plugins/ecutest/report/trf/TRFPublisherST.java
src/test/java/de/tracetronic/jenkins/plugins/ecutest/report/trf/TRFPublisherTest.java
src/test/java/de/tracetronic/jenkins/plugins/ecutest/test/TestFolderBuilderST.java
src/test/java/de/tracetronic/jenkins/plugins/ecutest/test/TestFolderBuilderTest.java
src/test/java/de/tracetronic/jenkins/plugins/ecutest/test/TestPackageBuilderST.java
src/test/java/de/tracetronic/jenkins/plugins/ecutest/test/TestPackageBuilderTest.java
src/test/java/de/tracetronic/jenkins/plugins/ecutest/test/TestProjectBuilderST.java
src/test/java/de/tracetronic/jenkins/plugins/ecutest/test/TestProjectBuilderTest.java
src/test/java/de/tracetronic/jenkins/plugins/ecutest/tool/StartETBuilderST.java
src/test/java/de/tracetronic/jenkins/plugins/ecutest/tool/StartETBuilderTest.java
src/test/java/de/tracetronic/jenkins/plugins/ecutest/tool/StartTSBuilderST.java
src/test/java/de/tracetronic/jenkins/plugins/ecutest/tool/StartTSBuilderTest.java
src/test/java/de/tracetronic/jenkins/plugins/ecutest/tool/StopETBuilderST.java
src/test/java/de/tracetronic/jenkins/plugins/ecutest/tool/StopETBuilderTest.java
src/test/java/de/tracetronic/jenkins/plugins/ecutest/tool/StopTSBuilderST.java
src/test/java/de/tracetronic/jenkins/plugins/ecutest/tool/StopTSBuilderTest.java
http://jenkins-ci.org/commit/ecutest-plugin/189cbd8a033bb7fb204ff7a68ad38cd0e5e90bf1
Log:
JENKINS-31999 added more tests

• avoid JENKINS-4409 in system tests
• removed unnecessary TestPluginManagerCleanUp
• added more unit tests
• added more system tests validating pipeline steps