Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-44924

pipeline groovy script - Sort a list with custom comparator or closure not sorting

       

      When using groovy script in the Pipeline Plugin, sorting a list using closure or a custom comparator does not work anymore.

       

      Steps to reproduce:

      1. create new item of type Pipeline
      2. In the Pipeline script add the following code
      #!groovy
      
      assert ["aa","bb","cc"] == ["aa","cc","bb"].sort { a, b -> a <=> b }
      1. Click Save
      2. Click Build Now
      3. Check the failed build:

      [Pipeline] End of Pipeline
      hudson.remoting.ProxyException: Assertion failed:

      assert ["aa","bb","cc"] == ["aa","cc","bb"].sort { a, b -> a <=> b }

      at org.codehaus.groovy.runtime.InvokerHelper.assertFailed(InvokerHelper.java:404)
      at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.assertFailed(ScriptBytecodeAdapter.java:650)
      at com.cloudbees.groovy.cps.impl.AssertBlock$ContinuationImpl.fail(AssertBlock.java:47)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
      at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
      at com.cloudbees.groovy.cps.Next.step(Next.java:83)
      at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:173)
      at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:162)
      at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:122)
      at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:261)
      at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:162)
      at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:19)
      at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:35)
      at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:32)
      at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
      at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:32)
      at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:174)
      at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:330)
      at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:82)
      at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:242)
      at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:230)
      at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
      at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:748)
      Finished: FAILURE

      Expected result is a sorted list:

      [aa, bb, cc]

       

       

          [JENKINS-44924] pipeline groovy script - Sort a list with custom comparator or closure not sorting

          Daniel Issac added a comment -

          Any updates on this.?I agree with Roman Sinyakov.This seems to be a serious issue which needs to be fixed.

          Daniel Issac added a comment - Any updates on this.?I agree with Roman Sinyakov.This seems to be a serious issue which needs to be fixed.

          Joerg Baeuerle added a comment - - edited

          Yes, I also agree. This has to be fixed. I've just spent two days of my life trying to figure out what was wrong with my pipeline script. My Groovy code worked perfectly in Jenkin's Script Console, but once put in a Jenkinsfile strange things happened. In my case it was not sort() but toSorted() which has the same problem, e.g. with the following code snippet

          def foo = ["hello","hi","hey"].toSorted { a, b -> a.length() <=> b.length() }
          println foo.toString()
          

          taken straight from the toSorted() specification in the Groovy Docs.

          In Jenkin's Script Console the output was

          [hi, hey, hello]
          

          which is correct. When run from a Jenkinsfile the build's output, however, was

          -1
          

           This is not acceptable and certainly not a "Minor" issue, because it costs a lot of time and money to hunt down these kinds of bugs!

          Joerg Baeuerle added a comment - - edited Yes, I also agree. This has to be fixed. I've just spent two days of my life trying to figure out what was wrong with my pipeline script. My Groovy code worked perfectly in Jenkin's Script Console, but once put in a Jenkinsfile strange things happened. In my case it was not sort() but toSorted() which has the same problem, e.g. with the following code snippet def foo = [ "hello" , "hi" , "hey" ].toSorted { a, b -> a.length() <=> b.length() } println foo.toString() taken straight from the toSorted() specification in the Groovy Docs . In Jenkin's Script Console the output was [hi, hey, hello] which is correct. When run from a Jenkinsfile the build's output, however, was -1  This is not acceptable and certainly not a "Minor" issue, because it costs a lot of time and money to hunt down these kinds of bugs!

          Pat Butkiewicz added a comment - - edited

          Just want to make sure I'm understanding this correctly. Jenkins has broken custom sorting in the Groovy language, expects people to write their Jenkinsfiles and Pipeline libraries in Groovy, and then marks this as a minor issue? Everyone on this thread agrees that's unacceptable, yes?

          Pat Butkiewicz added a comment - - edited Just want to make sure I'm understanding this correctly. Jenkins has broken custom sorting in the Groovy language, expects people to write their Jenkinsfiles and Pipeline libraries in Groovy, and then marks this as a minor issue? Everyone on this thread agrees that's unacceptable, yes?

          Andrew Bayer added a comment -

          If I could have viably fixed this, I would have already - the transformation done to Groovy code to be able to be durable makes a lot of things more or less impossible to get working correctly. We've dealt with as many of them as we could without having to basically write our own implementation of all of Groovy, but there are still edge cases (sorry, but this is a comparative edge case) where we just can't fix them. We're sorry, but this is what it is.

          Andrew Bayer added a comment - If I could have viably fixed this, I would have already - the transformation done to Groovy code to be able to be durable makes a lot of things more or less impossible to get working correctly. We've dealt with as many of them as we could without having to basically write our own implementation of all of Groovy, but there are still edge cases (sorry, but this is a comparative edge case) where we just can't fix them. We're sorry, but this is what it is.

          Neal Macdonald added a comment - - edited

          I couldn't get a pipeline script to work with map sort() at all until I came across this

          https://stackoverflow.com/questions/52946282/jenkins-sort-method-behaviour-not-as-expected. This might be a workaround for the issue above.

          Calling sort from a function with @NonCPS annotation eg:

          @NonCPS
          def sortExample(items) 
          {
              def itemsSorted = items.sort{ it['val'] }
              println(itemsSorted) 
          }
          
          node('targetnode')
          {
              def m = [ [name: 'abc', val: '123'], [name: 'zwe', val: '934'], [name: 'wxc', val: '789']]
              println m
              sortExample(m)
          }
          

          resulted in a full map sort dependent on the value of a specific key:

          [Pipeline] echo
          [{name=abc, val=123}, {name=zwe, val=934}, {name=wxc, val=789}]
          [Pipeline] echo
          [{name=abc, val=123}, {name=wxc, val=789}, {name=zwe, val=934}]
          

          Neal Macdonald added a comment - - edited I couldn't get a pipeline script to work with map sort() at all until I came across this https://stackoverflow.com/questions/52946282/jenkins-sort-method-behaviour-not-as-expected . This might be a workaround for the issue above. Calling sort from a function with @NonCPS annotation eg: @NonCPS def sortExample(items) { def itemsSorted = items.sort{ it['val'] } println(itemsSorted) } node('targetnode') { def m = [ [name: 'abc', val: '123'], [name: 'zwe', val: '934'], [name: 'wxc', val: '789']] println m sortExample(m) } resulted in a full map sort dependent on the value of a specific key: [Pipeline] echo [{name=abc, val=123}, {name=zwe, val=934}, {name=wxc, val=789}] [Pipeline] echo [{name=abc, val=123}, {name=wxc, val=789}, {name=zwe, val=934}]

          Lucas Cimon added a comment - - edited

          I found bit of helpful context about this issue : https://wiki.jenkins.io/display/JENKINS/Pipeline+CPS+method+mismatches#PipelineCPSmethodmismatches-Callingnon-CPS-transformedmethodswithCPS-transformedarguments

          Edit: captain obvious... I did not notice the link to this exact issue on the wiki

          Here is the workaround I used to sort a list of Map objects, based on a String field:

          pipeline {
              stages {
                  stage("Test") {
                      steps {
                          script {
                              def propList = [[name: 'B'], [name: 'A']]
                              sortByField(propList, 'name')
                              propList.eachWithIndex { entry, i ->
                                  println "${i}: ${entry}"
                              }
                          }
                      }
                  }
              }
          }
          @NonCPS
          static sortByField(list, fieldName) {
              list.sort{ it[fieldName] }
          }
           

          Lucas Cimon added a comment - - edited I found bit of helpful context about this issue : https://wiki.jenkins.io/display/JENKINS/Pipeline+CPS+method+mismatches#PipelineCPSmethodmismatches-Callingnon-CPS-transformedmethodswithCPS-transformedarguments Edit: captain obvious... I did not notice the link to this exact issue on the wiki Here is the workaround I used to sort a list of Map objects, based on a String field: pipeline { stages { stage( "Test" ) { steps { script { def propList = [[name: 'B' ], [name: 'A' ]] sortByField(propList, 'name' ) propList.eachWithIndex { entry, i -> println "${i}: ${entry}" } } } } } } @NonCPS static sortByField(list, fieldName) { list.sort{ it[fieldName] } }

          Ian Katz added a comment -

          If I could have viably fixed this, I would have already - the transformation done to Groovy code to be able to be durable makes a lot of things more or less impossible to get working correctly. We've dealt with as many of them as we could without having to basically write our own implementation of all of Groovy, but there are still edge cases (sorry, but this is a comparative edge case) where we just can't fix them.

          If the sort function can't work under any circumstances, shouldn't the correct behavior be to have it throw some exception rather than just pretend to work?  (Emphasis for meaning, not for tone.)

          I lost about a day's work here trying permutations of the following documented descriptions of array sort:

          I would have greatly preferred an exception that said "We're sorry, but this is what it is."  Instead I assumed that I wasn't supplying the correct arguments, mutating instead of returning a value, etc.  That was a very unpleasant rabbit hole.

          Can you talk more about what makes function impossible to support?  For example, is it related to a threaded library or some other implementation detail that prevents the built-in function from every being translated to CPS?  Is it a lack of support for the "spaceship operator"?  Or is it something inherent to the sort algorithm that would make it fail even if I tried to implement my own array sort function by hand?

          Ian Katz added a comment - If I could have viably fixed this, I would have already - the transformation done to Groovy code to be able to be durable makes a lot of things more or less impossible to get working correctly. We've dealt with as many of them as we could without having to basically write our own implementation of all of Groovy, but there are still edge cases (sorry, but this is a comparative edge case) where we just can't fix them. If the sort function can't work under any circumstances, shouldn't the correct behavior be to have it throw some exception rather than just pretend to work?  (Emphasis for meaning, not for tone.) I lost about a day's work here trying permutations of the following documented descriptions of array sort: http://docs.groovy-lang.org/next/html/documentation/working-with-collections.html#_sorting https://mrhaki.blogspot.com/2015/03/groovy-goodness-new-methods-to-sort-and.html https://mrhaki.blogspot.com/2011/09/groovy-goodness-sort-or-remove.html https://stackoverflow.com/a/20386474/2063546 I would have greatly preferred an exception that said "We're sorry, but this is what it is."  Instead I assumed that I wasn't supplying the correct arguments, mutating instead of returning a value, etc.  That was a very unpleasant rabbit hole. Can you talk more about what makes function impossible to support?  For example, is it related to a threaded library or some other implementation detail that prevents the built-in function from every being translated to CPS?  Is it a lack of support for the "spaceship operator"?  Or is it something inherent to the sort algorithm that would make it fail even if I tried to implement my own array sort function by hand?

          Jesse Glick added a comment -

          If you are using a reasonably recent version of workflow-cps it should have printed a warning with a link to a wiki page explaining this class of issue and workarounds.

          Jesse Glick added a comment - If you are using a reasonably recent version of workflow-cps it should have printed a warning with a link to a wiki page explaining this class of issue and workarounds.

          Ian Katz added a comment -

          I'm on 2.190.2 of Jenkins, with BlueOcean 1.21.0. Not sure how to find workflow-cps version.

          I didn't see a warning, but my point is that it should produce an error, not a warning. What is the justification for allowing me to call a function that you know in advance will not do its job?

          Ian Katz added a comment - I'm on 2.190.2 of Jenkins, with BlueOcean 1.21.0. Not sure how to find workflow-cps version. I didn't see a warning, but my point is that it should produce an error, not a warning. What is the justification for allowing me to call a function that you know in advance will not do its job?

          Jesse Glick added a comment -

          /pluginManager/installed; text console log in classic view, not Blue Ocean; and because the detector has false positives (it is very complicated).

          Jesse Glick added a comment - /pluginManager/installed ; text console log in classic view, not Blue Ocean; and because the detector has false positives (it is very complicated).

            Unassigned Unassigned
            persal79 Per Salomonsson
            Votes:
            29 Vote for this issue
            Watchers:
            43 Start watching this issue

              Created:
              Updated: