Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-26199

CpsScriptTest.evaluateShallBeCpsTransformed failures

    • Icon: Task Task
    • Resolution: Cannot Reproduce
    • Icon: Major Major
    • pipeline

      After several rounds of trial-and-error with 'head', and tag 'workflow-1.1' I'm still not sure why tests fail. Currently I'm using Ubuntu 14.04 / JDK 1.8.0_25 (I had open-jdk7), and I've installed all the stuff that tests threw at me (e.g. mercurial). I assigned port 8081 to my local Jenkins test installation, and removed ~/.jenkins (which mvn hpi:run uses if it exists). Normally I use Eclipse following the Jenkins plugin development recommendations, but I also tried using Netbeans (8.02) (which brought even more problems).

      Right now I still need to remove most of the tests from
      cps/src/test/groovy/org/jenkinsci/plugins/workflow/cps/CpsScriptTest.groovy in order to get the workflow-plugin running.

      Possibly there are system dependencies that my workstation doesn't meet. I know that the Java platform is supposed to be OS independent, but obviously it's not that simple. It would be very helpful if there were a list of requirements for the build- and test environment (Linux, BSD, Mac, Windows, distribution, installed software ... ).

          [JENKINS-26199] CpsScriptTest.evaluateShallBeCpsTransformed failures

          Jesse Glick added a comment -

          There are no specific build requirements I know of, other than that many tests are known not to run on Windows (because they rely on the sh step and the like). Personally I use NetBeans with JDK 8 on Ubuntu 14.10.

          So what are your test failures? (And do you really need to run all tests yourself, or would mvn -DskipTests clean install suffice? If you intend to submit changes, just file a pull request and your changes will be tested automatically.)

          ~/.jenkins/ is irrelevant for development; mvn hpi:run does not use it (it uses ./work), and tests using JenkinsRule certainly do not use it.

          Jesse Glick added a comment - There are no specific build requirements I know of, other than that many tests are known not to run on Windows (because they rely on the sh step and the like). Personally I use NetBeans with JDK 8 on Ubuntu 14.10. So what are your test failures? (And do you really need to run all tests yourself, or would mvn -DskipTests clean install suffice? If you intend to submit changes, just file a pull request and your changes will be tested automatically.) ~/.jenkins/ is irrelevant for development; mvn hpi:run does not use it (it uses ./work ), and tests using JenkinsRule certainly do not use it.

          You're using Ubuntu, too - that's great (and safes me a lot of money, and frustration :-]) I'll give Netbeans one more try, the Stapler plugin makes me curious.

          First, here is the CpsScriptTest failure I get:

          ```
          Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 4.815 sec <<< FAILURE! - in org.jenkinsci.plugins.workflow.cps.CpsScriptTest
          evaluateShallBeCpsTransformed(org.jenkinsci.plugins.workflow.cps.CpsScriptTest) Time elapsed: 2.715 sec <<< FAILURE!
          org.codehaus.groovy.runtime.powerassert.PowerAssertionError: assert future != null

           

          null false
          at org.codehaus.groovy.runtime.InvokerHelper.assertFailed(InvokerHelper.java:386)
          at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.assertFailed(ScriptBytecodeAdapter.java:658)
          at org.jenkinsci.plugins.workflow.cps.CpsScriptTest.evaluateShallBeCpsTransformed(CpsScriptTest.groovy:74)

          Results :

          Failed tests:
          CpsScriptTest.evaluateShallBeCpsTransformed:74 assert future != null

           

          null false

          Tests run: 21, Failures: 1, Errors: 0, Skipped: 0
          ```

          The test code in tag 'workflow-1.1' was different, but it failed, too.

          Second, you may have guessed that I'm a Jenkins-dev newbie (comming from the world of embedded control). I assumed that I better don't issue any pull-request before I know that my code doesn't break tests (which I can't when I switch off tests . Assumptions are often wrong, but I read all the docs on development carefully, and it they only told me to run 'mvn clean install', there is no word about skipping the tests. Of course, you must assume a certain 'dev proficiency', but sometimes such things make the learning curve steep.

          Third, try adding a directory ~/.jenkins without write access, and run 'mvn clean install'. On my machine this makes (many) tests fail. On the other hand, if that folder doesn't exist, it's there after the test are done.

          This is what I find there:

          ```
          thomas@thomas-W500:~$ ls .jenkins/
          cache
          thomas@thomas-W500:~$ ls .jenkins/cache/
          jars
          thomas@thomas-W500:~$ ls .jenkins/cache/jars/
          04 1A 2C 3A 4B 50 53 63 68 76 86 94 A7 D2 DB F5
          09 1F 37 43 4F 52 61 65 69 78 8D 9B C2 D5 F2 FE
          thomas@thomas-W500:~$ ls .jenkins/cache/jars/04
          66AB2CDE8F87045B932F61151D38B9.jar
          ```

          Maybe that's because I run 'mvn install' from the workflow-plugin root, and not from 'aggregator', but in my opinion that's not obvious.

          It would also have been helpful to know, where I should run 'mvn hpi:run' (maybe it's obvious, but the workflow plugin is a 'bit' more complex than most examples , and I have still to figure out how to do this with netbeans).

          Of course, most of this isn't a problem of the workflow-plugin, but every bit of information helps.

          Thomas Goeppel added a comment - You're using Ubuntu, too - that's great (and safes me a lot of money, and frustration :-]) I'll give Netbeans one more try, the Stapler plugin makes me curious. First, here is the CpsScriptTest failure I get: ``` Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 4.815 sec <<< FAILURE! - in org.jenkinsci.plugins.workflow.cps.CpsScriptTest evaluateShallBeCpsTransformed(org.jenkinsci.plugins.workflow.cps.CpsScriptTest) Time elapsed: 2.715 sec <<< FAILURE! org.codehaus.groovy.runtime.powerassert.PowerAssertionError: assert future != null   null false at org.codehaus.groovy.runtime.InvokerHelper.assertFailed(InvokerHelper.java:386) at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.assertFailed(ScriptBytecodeAdapter.java:658) at org.jenkinsci.plugins.workflow.cps.CpsScriptTest.evaluateShallBeCpsTransformed(CpsScriptTest.groovy:74) Results : Failed tests: CpsScriptTest.evaluateShallBeCpsTransformed:74 assert future != null   null false Tests run: 21, Failures: 1, Errors: 0, Skipped: 0 ``` The test code in tag 'workflow-1.1' was different, but it failed, too. Second, you may have guessed that I'm a Jenkins-dev newbie (comming from the world of embedded control). I assumed that I better don't issue any pull-request before I know that my code doesn't break tests (which I can't when I switch off tests . Assumptions are often wrong, but I read all the docs on development carefully, and it they only told me to run 'mvn clean install', there is no word about skipping the tests. Of course, you must assume a certain 'dev proficiency', but sometimes such things make the learning curve steep. Third, try adding a directory ~/.jenkins without write access, and run 'mvn clean install'. On my machine this makes (many) tests fail. On the other hand, if that folder doesn't exist, it's there after the test are done. This is what I find there: ``` thomas@thomas-W500:~$ ls .jenkins/ cache thomas@thomas-W500:~$ ls .jenkins/cache/ jars thomas@thomas-W500:~$ ls .jenkins/cache/jars/ 04 1A 2C 3A 4B 50 53 63 68 76 86 94 A7 D2 DB F5 09 1F 37 43 4F 52 61 65 69 78 8D 9B C2 D5 F2 FE thomas@thomas-W500:~$ ls .jenkins/cache/jars/04 66AB2CDE8F87045B932F61151D38B9.jar ``` Maybe that's because I run 'mvn install' from the workflow-plugin root, and not from 'aggregator', but in my opinion that's not obvious. It would also have been helpful to know, where I should run 'mvn hpi:run' (maybe it's obvious, but the workflow plugin is a 'bit' more complex than most examples , and I have still to figure out how to do this with netbeans). Of course, most of this isn't a problem of the workflow-plugin, but every bit of information helps.

          Jesse Glick added a comment -

          Stapler plugins are available for both NetBeans and IntelliJ. Either are well supported for Jenkins plugin development. Eclipse is not well supported.

          I did not write CpsScriptTest but perhaps kohsuke can take a look at that failure. Anyway, a single test failure is not much cause for concern, when there are a number of tests that fail on occasion, perhaps due to race conditions. You can just file a pull request and a Jenkins builder will automatically run all tests against your changes and report any failures, which plugin maintainers would evaluate to see if they are significant or not. I rarely run all tests in the Workflow plugin myself, as it would take much too long; I would only run tests I think might be related to changes I am making.

          ~/.jenkins/cache/ is JENKINS-18578.

          For Workflow, hpi:run should be run inside aggregator, which from NetBeans you can do via just pressing Run Project when you have the Jenkins developer plugin installed: https://github.com/jenkinsci/workflow-plugin/blob/master/README.md#source-organization

          By the way, if you are unfamiliar with Jenkins development, I will warn you that many (though not all) parts of Workflow are especially tough to get into, compared to most other Jenkins plugins.

          Jesse Glick added a comment - Stapler plugins are available for both NetBeans and IntelliJ. Either are well supported for Jenkins plugin development. Eclipse is not well supported. I did not write CpsScriptTest but perhaps kohsuke can take a look at that failure. Anyway, a single test failure is not much cause for concern, when there are a number of tests that fail on occasion, perhaps due to race conditions. You can just file a pull request and a Jenkins builder will automatically run all tests against your changes and report any failures, which plugin maintainers would evaluate to see if they are significant or not. I rarely run all tests in the Workflow plugin myself, as it would take much too long; I would only run tests I think might be related to changes I am making. ~/.jenkins/cache/ is JENKINS-18578 . For Workflow, hpi:run should be run inside aggregator , which from NetBeans you can do via just pressing Run Project when you have the Jenkins developer plugin installed: https://github.com/jenkinsci/workflow-plugin/blob/master/README.md#source-organization By the way, if you are unfamiliar with Jenkins development, I will warn you that many (though not all) parts of Workflow are especially tough to get into, compared to most other Jenkins plugins.

          You're right, the workflow-plugin is a complex piece of software, and it gets obvious in the test automation (kudos to Kohsuke Kawaguchi, and to you!). I now understand, that the test environment is tricky, and ironing out race-conditions isn't easy. If reporting such issues helps, please let me know.

          However, I'd like to do some experiments with the code, and maybe the things I'd like to implement are out of my reach. In any case, I won't tamper with the CPS/Groovy magic at the heart of the plugin, and if I get anything of general interest, I'll use a pull-request for testing, like you suggested.

          Thanks for all the useful hints, and thanks for pointing out JENKINS-18578 - it has an impact on one of my use cases for Jenkins, and I guess you just saved me days of work.

          Thomas Goeppel added a comment - You're right, the workflow-plugin is a complex piece of software, and it gets obvious in the test automation (kudos to Kohsuke Kawaguchi, and to you!). I now understand, that the test environment is tricky, and ironing out race-conditions isn't easy. If reporting such issues helps, please let me know. However, I'd like to do some experiments with the code, and maybe the things I'd like to implement are out of my reach. In any case, I won't tamper with the CPS/Groovy magic at the heart of the plugin, and if I get anything of general interest, I'll use a pull-request for testing, like you suggested. Thanks for all the useful hints, and thanks for pointing out JENKINS-18578 - it has an impact on one of my use cases for Jenkins, and I guess you just saved me days of work.

          Jesse Glick added a comment -

          Yes please report test failures you find. Some are known and the tests are marked @RandomlyFails until they can be written better; see JENKINS-25975.

          On reflection it occurs to me that much of the workflow-cps plugin is not Jenkins-specific code, so being relatively unfamiliar with Jenkins would not matter much. It is just difficult code to follow, period. Some other portions of the code, especially the workflow-job module, rely much more heavily on Jenkins internals, but have more straightforward control flow.

          Jesse Glick added a comment - Yes please report test failures you find. Some are known and the tests are marked @RandomlyFails until they can be written better; see JENKINS-25975 . On reflection it occurs to me that much of the workflow-cps plugin is not Jenkins-specific code, so being relatively unfamiliar with Jenkins would not matter much. It is just difficult code to follow, period. Some other portions of the code, especially the workflow-job module, rely much more heavily on Jenkins internals, but have more straightforward control flow.

          Jesse Glick added a comment -

          Hope this was fixed since reported. If not, please reopen with details.

          Jesse Glick added a comment - Hope this was fixed since reported. If not, please reopen with details.

            Unassigned Unassigned
            tg9541 Thomas Goeppel
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: