Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46988

String.eachLine only reads first line

    XMLWordPrintable

Details

    Description

      As with JENKINS-26481 the eachLine-method on String only works for the first iteration.

      Using version 3.39

      We use a cleanup-pipeline to find all running docker-containers without a corresponding feature-branch (deleted after merge).

      stage('clean') {
       def branches = []
       // extract available branches from git
       sh (returnStdout: true, script: "ssh-agent bash -c 'ssh-add /var/lib/jenkins/.ssh/id_rsa &>/dev/null; git ls-remote --heads --refs ssh://git@myrepo/project.git' | cut -f 2")
        .eachLine { branches << it }
       // extract all containers (including stopped)
       def containers = []
       sh (returnStdout: true, script: "docker ps -a --format '{{.Names}}' --filter name=project")
        .trim()
        .eachLine { containers << it }
       println(containers) // <---- only prints first container
       println(branches) // <---- only prints first branch
      
       //stop containers for non existing branches...
       containers.each{ containername ->
        if(branches.findAll({branch -> branch.contains(containername)}).isEmpty()){
         println("trying to stop ${containername}")
         //sh ("docker stop ${containername} || true") // container might already be stopped
         println("removing ${containername}")
         //sh ("docker rm ${containername}")
        }
       }
      }

      As a workaround replace eachLine with split('\n').each { ... }

      Attachments

        Activity

          Also wasted time one this issue, but thanks to this bug report, quickly switched to workaround.

          dmitryb Dmitry Bigunyak added a comment - Also wasted time one this issue, but thanks to this bug report, quickly switched to workaround.
          bjmgeek Brian Minton added a comment - - edited

          Confirming that this is still happening as of Jenkins 2.319.1.  I agree with tonyhoylerps that the CPS mismatch link is not helpful.

          bjmgeek Brian Minton added a comment - - edited Confirming that this is still happening as of Jenkins 2.319.1.  I agree with tonyhoylerps that the CPS mismatch link is not helpful.
          markewaite Mark Waite added a comment -

          The Jenkins project recommends that these types of complex operations be performed with an sh step that calls the program of your choice. You can use groovy in that sh step or whatever other language you prefer. By placing the complex operation in an sh step, it can be executed on an agent instead of on the controller. By placing the complex operation in an sh step, it can use whatever language and version you prefer.

          markewaite Mark Waite added a comment - The Jenkins project recommends that these types of complex operations be performed with an sh step that calls the program of your choice. You can use groovy in that sh step or whatever other language you prefer. By placing the complex operation in an sh step, it can be executed on an agent instead of on the controller. By placing the complex operation in an sh step, it can use whatever language and version you prefer.
          jorhett Jo Rhett added a comment -

          The Jenkins project recommends that these types of complex operations be performed with an sh step that calls the program of your choice.

          Thank you for an utterly irrelevant copypasta response. The example shown is an sh step that calls the program of their choice, Mark. So your post just wasted everyone's time because it's not relevant or helpful in any way.

          jorhett Jo Rhett added a comment - The Jenkins project recommends that these types of complex operations be performed with an sh step that calls the program of your choice. Thank you for an utterly irrelevant copypasta response. The example shown is an sh step that calls the program of their choice, Mark. So your post just wasted everyone's time because it's not relevant or helpful in any way.
          markewaite Mark Waite added a comment -

          jorhett I don't think you understood the message that I was trying to convey.

          The script provided in the bug description uses the Jenkins agent to perform an sh call to list the branches and uses the Jenkins agent to perform an sh call to list the containers. It then uses the Jenkins controller to iterate over the list of containers and the list of branches to decide which containers should be stopped. Then it calls sh to use the Jenkins agent to stop the containers and remove them.

          I think it is simpler to develop, diagnose, and debug if that entire sequence of operations (list branches, list containers, choose containers to stop, stop containers) is performed on the agent inside an sh step that calls a script that runs entirely on the agent. The script that I wrote to do that type of operation in Python looks like this:

          #!/usr/bin/python3
          
          import subprocess
          import sys
          
          #-----------------------------------------------------------------------
          
          def get_branches():
              ls_remote_lines = subprocess.check_output(['git', 'ls-remote', '--heads', '--refs', '--quiet']).decode('utf-8').split('\n')
              branches = []
              for line in ls_remote_lines:
                  if 'refs/' in line:
                      sha1, branch_name = line.split('\t')
                      branches.append(branch_name.replace('refs/heads/', ''))
              return branches
          
          #-----------------------------------------------------------------------
          
          def get_containers():
              containers_lines = subprocess.check_output(['docker', 'ps', '-a', '--format', '{{.Names}}']).decode('utf-8').split('\n')
              return containers_lines[:-1] # discard trailing empty line
          
          #-----------------------------------------------------------------------
          
          def stop_container(container):
              print("docker stop " + container)
          
          #-----------------------------------------------------------------------
          
          def cleanup_pipeline(args = []):
              branches = get_branches()
              # print('==== Branches ====')
              # for branch in branches:
              #     print(branch)
          
              containers = get_containers()
              # print('==== Containers ====')
              # for container in containers:
              #     print(container)
          
              # Stop containers for non-existent branches
              for container in containers:
                  found = False
                  for branch in branches:
                      if branch in container:
                          found = True
                  if not found:
                      stop_container(container)
          
          #-----------------------------------------------------------------------
          
          if __name__ == "__main__": cleanup_pipeline(sys.argv[1:])
          

          The Jenkins project recommends keeping Pipeline scripts simple and pushing as much logic as is feasible into scripts that run on the agent. See the "How to use Jenkins less" talk from Jesse Glick and the "Use just enough pipeline" blog post by Darin Pope and me.

          I truly was trying to provide useful information with my comment. My apologies if it was not received as such.

          markewaite Mark Waite added a comment - jorhett I don't think you understood the message that I was trying to convey. The script provided in the bug description uses the Jenkins agent to perform an sh call to list the branches and uses the Jenkins agent to perform an sh call to list the containers. It then uses the Jenkins controller to iterate over the list of containers and the list of branches to decide which containers should be stopped. Then it calls sh to use the Jenkins agent to stop the containers and remove them. I think it is simpler to develop, diagnose, and debug if that entire sequence of operations (list branches, list containers, choose containers to stop, stop containers) is performed on the agent inside an sh step that calls a script that runs entirely on the agent. The script that I wrote to do that type of operation in Python looks like this: #!/usr/bin/python3 import subprocess import sys #----------------------------------------------------------------------- def get_branches(): ls_remote_lines = subprocess.check_output(['git', 'ls-remote', '--heads', '--refs', '--quiet']).decode('utf-8').split('\n') branches = [] for line in ls_remote_lines: if 'refs/' in line: sha1, branch_name = line.split('\t') branches.append(branch_name.replace('refs/heads/', '')) return branches #----------------------------------------------------------------------- def get_containers(): containers_lines = subprocess.check_output(['docker', 'ps', '-a', '--format', '{{.Names}}']).decode('utf-8').split('\n') return containers_lines[:-1] # discard trailing empty line #----------------------------------------------------------------------- def stop_container(container): print("docker stop " + container) #----------------------------------------------------------------------- def cleanup_pipeline(args = []): branches = get_branches() # print('==== Branches ====') # for branch in branches: # print(branch) containers = get_containers() # print('==== Containers ====') # for container in containers: # print(container) # Stop containers for non-existent branches for container in containers: found = False for branch in branches: if branch in container: found = True if not found: stop_container(container) #----------------------------------------------------------------------- if __name__ == "__main__": cleanup_pipeline(sys.argv[1:]) The Jenkins project recommends keeping Pipeline scripts simple and pushing as much logic as is feasible into scripts that run on the agent. See the "How to use Jenkins less" talk from Jesse Glick and the "Use just enough pipeline" blog post by Darin Pope and me. I truly was trying to provide useful information with my comment. My apologies if it was not received as such.

          People

            Unassigned Unassigned
            lostiniceland Marc Schlegel
            Votes:
            32 Vote for this issue
            Watchers:
            26 Start watching this issue

            Dates

              Created:
              Updated: