7f979ccc broke PLUGINS_DIR feature. If I run test against some plugins I do not want test suite to change the versions under test in any way.

Hmm, isn't @WithPlugins there to create the subject under test (SUT)? Seems that actually the PLUGIN_DIR feature is broken? If I follow your argumentation, then no plugin should be installed at all using @WithPlugins.

Besides, what version of core or plugins will be is used should be configured per test execution and not by test creator.

And how do I do that? When I start my tests using 'mvn test -DPluginTests' I can't provide the plugin versions to load. The SUT will get populated automatically.

You can still create tests for newer plugins (using the same annotation). The difference is that the test will be skipped if older version is used, instead of silently updating the plugin and creating false impression of successful result.

What is your use case?

See comment above: Jenkins war contains prepackaged plug-ins that are old.

Isn't there a way to get both behaviors? Not changing the PLUGIN_DIR option AND allowing to test new plugins without creating an explicit fixture.

Hmm, isn't @WithPlugins there to create the subject under test (SUT)? Seems that actually the PLUGIN_DIR feature is broken? If I follow your argumentation, then no plugin should be installed at all using @WithPlugins.

PLUGINS_DIR is useful for upgrade tests, verify certain jenkins build and plugin builds work together. Perhaps there is a better way to achieve the same.

All I propose is, if there is the plugin already installed (in version that satisfies version constraint) we should keep it as it is. Otherwise install latest version, not the exact version from @WithPlugins. This effectively changes the semantics of version to minimal version.

And how do I do that? When I start my tests using 'mvn test -DPluginTests' I can't provide the plugin versions to load. The SUT will get populated automatically.

There is no convenient way to do that for now, aside from putting {{.hpi}}s to PLUGINS_DIR first.

Jenkins war contains prepackaged plug-ins that are old.

Sure, but if we are shipping weekly/stable release with buggy bundled plugin tests should fail. If we allow test creator to download plugin version with fix we are hiding the fact we ship broken plugin.

Though, I agree it make sense to run the tests with latest released versions too. Perhaps it should be configurable.

All I propose is, if there is the plugin already installed (in version that satisfies version constraint) we should keep it as it is. Otherwise install latest version, not the exact version from @WithPlugins. This effectively changes the semantics of version to minimal version.

Good, then we have the same use case. But your implemented algorithm does work differently or am I missing something?
Your algorithm:

if not installed
  install latest version from update center
  if this version is older than required
    fail the test
else
  if this version is older than required
    skip the test

Our required algorithm:

if not installed
  install latest version from update center
  if this version is older than required
    fail the test
else
  if this version is older than required
    install latest version from update center
    if this version is older than required
      fail the test

Otherwise I don't see a way to load a plug-in with a newer version than the bundled version.

Agree, both of the approaches makes sense. My implementation is supposed to be equivalent to pseudocode in description.

We need configuration option for either always use latest version when @WithPlugins is used or upgrade all plugins to latest versions. I am undecided on which one to use.

If I understand the following code correctly, then PLUGIN_DIR is newer overwritten. Just the copy will be overwritten: https://github.com/jenkinsci/acceptance-test-harness/blob/master/src/main/java/org/jenkinsci/test/acceptance/controller/LocalController.java#L148

So I suggest to add a new environment variable, e.g. NEVER_REPLACE_EXISTING_PLUGINS.

What do you think? From a users perspective, I think the default case for a test with @version annotation should be to install that version if it exists in an older version.

> So I suggest to add a new environment variable, e.g. NEVER_REPLACE_EXISTING_PLUGINS.

That would work.

> From a users perspective, I think the default case for a test with @version annotation should be to install that version if it exists in an older version.

I do not like the idea of installing the exact version in @WithPlugins annotation. If the plugin is not present and we are about to install it, we should pick whatever update center suggest as a default version (IOW latest version available). Or are you suggesting some new concept of @Version.

IMO it definitely should not downgrade if already installed (that is how it worked before my fix).

I do not like the idea of installing the exact version in @WithPlugins annotation. If the plugin is not present and we are about to install it, we should pick whatever update center suggest as a default version (IOW latest version available). Or are you suggesting some new concept of @Version.

IMO it definitely should not downgrade if already installed (that is how it worked before my fix).

Sorry, my comment was not precise: I don't want to downgrade an existing version. If a newer version is already installed than installation should be skipped. If installed version is too old, than the latest version should be installed.

Ok, so we have an agreement: JENKINS-23093

Ok, then I'm closing this. Do we revert your commits until JENKINS-23093 is fixed?

Code changed in jenkins
User: Ulli Hafner
Path:
docs/CONTROLLER.md
src/main/java/org/jenkinsci/test/acceptance/junit/WithPlugins.java
http://jenkins-ci.org/commit/acceptance-test-harness/1aedbbcbec207d98a86848648aa45d50abe9f8f3
Log:
[FIXED JENKINS-23093]JENKINS-23087 Fixed WithPlugins annotation.
Now existing plugin will be overwritten with newer version if env var
NEVER_REPLACE_EXISTING_PLUGINS is not set.