Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-44442

Result of calt call is not machine friendly

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      My issue is that I am calling state.highstate (jenkins groovy code:

      salt authtype: 'pam', clientInterface: local(arguments: '"-state-output=changes -v"', function: 'state.highstate', target: 'acceptanceGroup', targettype: 'nodegroup'), credentialsId: 'SOMEUUID', saveFile: true, servername: 'http://salt.domain.sk:8000'

      ) and I am not able to programatically get result of highstates. Even, when I pass "--hard-crash" argument, result is always meaningless for some computer processing. I can hardcode search for keywords as "stacktrace", "Fail", "False", "jinja error". Is it only way? I have expectd main reason why to bother with salt api and salt plugin for jenkins that I will get proper machine friendly results opposed to call to shell.

       

      Is there any possibility to get result of highstate or at least exit code of salt-call invocation from minion?

       

        Attachments

          Activity

          Hide
          lubo_varga_nike Lubo Varga added a comment -

          Other suggested possibility is to pass --retcode-passthrough{{}} (documentation https://docs.saltstack.com/en/latest/ref/cli/salt-call.html), but I am not able to find, how is retcode calculated when salt-call is done on more than one minion.

          Show
          lubo_varga_nike Lubo Varga added a comment - Other suggested possibility is to pass --retcode-passthrough {{}} (documentation https://docs.saltstack.com/en/latest/ref/cli/salt-call.html ), but I am not able to find, how is retcode calculated when salt-call is done on more than one minion.
          Hide
          mchugh19 Christian McHugh added a comment - - edited

          The jenkins plugin will send the requested commands to the salt-api, and once complete, attempts to validate for failures. When you say that you are 

          not able to programatically get result of highstates

          do you mean that you would like to capture the output from the highstate, or that the validation step is not accurate (either failing for a successful highstate, or not failing the build on highstate failure)?

          If the jenkins plugin's validation needs to be adjusted based on highstate output, could you please send over the hightstate json return? 

           

          Additionally, salt-call is local to the minion. The salt-api communicates with the salt-master to which the minions then communicate with through the salt event bus. So if you are looking up command line syntax, it will be the salt cli, not the salt-call that is more equivalent. Having said that, the salt-api does not necessarily return the retcode for all steps in a highstate (see https://github.com/saltstack/salt/issues/37043)

          Show
          mchugh19 Christian McHugh added a comment - - edited The jenkins plugin will send the requested commands to the salt-api, and once complete, attempts to validate for failures. When you say that you are  not able to programatically get result of highstates do you mean that you would like to capture the output from the highstate, or that the validation step is not accurate (either failing for a successful highstate, or not failing the build on highstate failure)? If the jenkins plugin's validation needs to be adjusted based on highstate output, could you please send over the hightstate json return?    Additionally, salt-call is local to the minion. The salt-api communicates with the salt-master to which the minions then communicate with through the salt event bus. So if you are looking up command line syntax, it will be the salt cli, not the salt-call that is more equivalent. Having said that, the salt-api does not necessarily return the retcode for all steps in a highstate (see https://github.com/saltstack/salt/issues/37043 )
          Hide
          lubo_varga_nike Lubo Varga added a comment -

          Output seems fine, I am saving it to file and than parsing like this:

                  stage('Tryout') {
                      ansiColor(colorMapName: 'xterm') {
                          println 'Goint to call \u001B[1msalt state.highstate\u001B[0m through salt api (rest api).'
                          salt authtype: 'pam', clientInterface: local(arguments: '"--state-output=changes"', function: 'state.highstate', target: 'accgrp', targettype: 'nodegroup'), credentialsId: 'SomeUUID', saveFile: true, servername: 'http://salt.example.xx:8000'
                          // this will save json to file. Json is like this:
                          // [{"minion1":{},"minion2":{},"minion3":{}}]
                          def json = readJSON file: 'saltOutput.json'
                          for ( int i = 0; i < json[0].size(); i++ ) {
                              // hacktacular String() cloning to avoid NotSerializableException; also
                              // hacktacular Map > Set > Array morph to enable C-style looping
                              final key = json[0].entrySet().toArray()[i].key
                              final value = json[0].entrySet().toArray()[i].value
                              // do stuff with key and value...
                              println "Minion \u001B[7m${key}\u001B[0m:\n${value}\n"
                          }
                      }
                  }

          Than I "accidentally" make wrong edit in pillar (just mess yaml format) and run pipeline on group of three minions. Output to console is like this:

          Minion acc-3.example.sk:
          The minion function caused an exception: Traceback (most recent call last):
            File "/usr/lib/python2.6/site-packages/salt/minion.py", line 1445, in _thread_return
              return_data = executor.execute()
            File "/usr/lib/python2.6/site-packages/salt/executors/direct_call.py", line 28, in execute
              return self.func(*self.args, **self.kwargs)
            File "/usr/lib/python2.6/site-packages/salt/modules/state.py", line 815, in highstate
              err += __pillar__['_errors']
            File "/usr/lib/python2.6/site-packages/salt/utils/context.py", line 211, in __getitem__
              return self._dict()[key]
          KeyError: '_errors'
          
          
          [Pipeline] echo
          Minion acc-1.example.sk:
          [Pillar failed to render with the following messages:, Rendering SLS 'backend.acc' failed. Please see master log for details.]
          
          [Pipeline] echo
          Minion acc-2.example.sk:
          [Pillar failed to render with the following messages:, Failed to load ext_pillar git: 'git init' returned exit status 128: error: could not lock config file /var/cache/salt/master/pillar_gitfs/6e9e37db81f85ra48403fedd6c3c220f4b02ae85848s8/.git/config: File exists
          fatal: could not set 'core.repositoryformatversion' to '0']

          What you see is placed in json result originaly. I see as problem, that this step passes. Build is green and there missing/wrong/partial validation of actual result of highstate command.

          According issue link which you have posted (btw, wrong url is in link. probably url and link name are swapped), it seems that validation is not possible in "machine readable" way. I mean that making list of words which cause failure of result is not machine friendly way. I expect (from salt-api; i.e. not from your plugin) to return "status code"/"exit code"/result from each minion and put it to resulting json (something like that is proposed in given salt issue). Than your plugin would have easy job, to evaluate if all minions returned "no error happened" result and act according that.

          Now I see that problem origin is in salt-api (or salt command directly).

          Would you mind add workaround solution to your plugin, which would help other users? I would propose to have one more parameter for plugin: throwOnProblem: Boolean. It will optionally activate behaviour that if result seems that some minion has returned "failed" result, it will throw an exception with list of minions and corresponding errors. I would like to provide you some cases of "error" responses. This way, I will not code same things like a few other people does.

          PS: our current "failure detection" is using shell (not salt-api) like this:

          #partial bash code:
          
          salt --state-verbose=False --state-output=mixed --force-color -N "$minions" state.highstate "$pillar" > $output_file 2>&1
          if grep -q "Failed:[ ]*[1-9][0-9]*" $output_file || [ "$salt_result" -ne 0 ] ; then
           echo "$minions NOT provisioned in $runtime seconds. Saltstack output:"
           cat $output_file
           rm -f $output_file
           exit 1
          else
           if grep -q "changed" $output_file ; then
            echo "$minions PROVISIONED in $runtime seconds. Changes:"
            cat $output_file
           else
            echo "$minions provisioned with NO CHANGES in $runtime seconds."
           fi
           rm -f $output_file
          fi
          Show
          lubo_varga_nike Lubo Varga added a comment - Output seems fine, I am saving it to file and than parsing like this:         stage( 'Tryout' ) {             ansiColor(colorMapName: 'xterm' ) {                 println 'Goint to call \u001B[1msalt state.highstate\u001B[0m through salt api ( rest api).'                 salt authtype: 'pam' , clientInterface: local(arguments: ' "--state-output=changes" ' , function: 'state.highstate' , target: 'accgrp' , targettype: 'nodegroup' ), credentialsId: 'SomeUUID' , saveFile: true , servername: 'http: //salt.example.xx:8000'                 // this will save json to file. Json is like this :                 // [{ "minion1" :{}, "minion2" :{}, "minion3" :{}}]                 def json = readJSON file: 'saltOutput.json'                 for ( int i = 0; i < json[0].size(); i++ ) {                     // hacktacular String () cloning to avoid NotSerializableException; also                     // hacktacular Map > Set > Array morph to enable C-style looping                     final key = json[0].entrySet().toArray()[i].key                     final value = json[0].entrySet().toArray()[i].value                     // do stuff with key and value...                     println "Minion \u001B[7m${key}\u001B[0m:\n${value}\n"                 }             }         } Than I "accidentally" make wrong edit in pillar (just mess yaml format) and run pipeline on group of three minions. Output to console is like this: Minion acc-3.example.sk: The minion function caused an exception: Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/salt/minion.py" , line 1445, in _thread_return return_data = executor.execute() File "/usr/lib/python2.6/site-packages/salt/executors/direct_call.py" , line 28, in execute return self.func(*self.args, **self.kwargs) File "/usr/lib/python2.6/site-packages/salt/modules/state.py" , line 815, in highstate err += __pillar__[ '_errors' ] File "/usr/lib/python2.6/site-packages/salt/utils/context.py" , line 211, in __getitem__ return self._dict()[key] KeyError: '_errors' [Pipeline] echo Minion acc-1.example.sk: [Pillar failed to render with the following messages:, Rendering SLS 'backend.acc' failed. Please see master log for details.] [Pipeline] echo Minion acc-2.example.sk: [Pillar failed to render with the following messages:, Failed to load ext_pillar git: 'git init' returned exit status 128: error: could not lock config file / var /cache/salt/master/pillar_gitfs/6e9e37db81f85ra48403fedd6c3c220f4b02ae85848s8/.git/config: File exists fatal: could not set 'core.repositoryformatversion' to '0' ] What you see is placed in json result originaly. I see as problem, that this step passes. Build is green and there missing/wrong/partial validation of actual result of highstate command. According issue link which you have posted (btw, wrong url is in link. probably url and link name are swapped), it seems that validation is not possible in "machine readable" way. I mean that making list of words which cause failure of result is not machine friendly way. I expect (from salt-api; i.e. not from your plugin) to return "status code"/"exit code"/result from each minion and put it to resulting json (something like that is proposed in given salt issue). Than your plugin would have easy job, to evaluate if all minions returned "no error happened" result and act according that. Now I see that problem origin is in salt-api (or salt command directly). Would you mind add workaround solution to your plugin, which would help other users? I would propose to have one more parameter for plugin: throwOnProblem: Boolean. It will optionally activate behaviour that if result seems that some minion has returned "failed" result, it will throw an exception with list of minions and corresponding errors. I would like to provide you some cases of "error" responses. This way, I will not code same things like a few other people does. PS: our current "failure detection" is using shell (not salt-api) like this: #partial bash code: salt --state-verbose=False --state-output=mixed --force-color -N "$minions" state.highstate "$pillar" > $output_file 2>&1 if grep -q "Failed:[ ]*[1-9][0-9]*" $output_file || [ "$salt_result" -ne 0 ] ; then  echo "$minions NOT provisioned in $runtime seconds. Saltstack output:"  cat $output_file  rm -f $output_file  exit 1 else   if grep -q "changed" $output_file ; then   echo "$minions PROVISIONED in $runtime seconds. Changes:"   cat $output_file   else   echo "$minions provisioned with NO CHANGES in $runtime seconds."  fi  rm -f $output_file fi
          Hide
          mchugh19 Christian McHugh added a comment -

          It sounds like there are a couple of cases where failures are not being detected by the Jenkins plugin. Could you please send over the output from the salt-api, and I'll get that fixed.

          Show
          mchugh19 Christian McHugh added a comment - It sounds like there are a couple of cases where failures are not being detected by the Jenkins plugin. Could you please send over the output from the salt-api, and I'll get that fixed.
          Hide
          lubo_varga_nike Lubo Varga added a comment - - edited

          case one

          error case is, when in output file is saved this: 

          [{}]
          

          But on salt server in file /var/log/salt/api I see timeout errors:

          2017-05-26 08:43:27,811 [tornado.access                                     ][INFO    ][103027] 200 POST /login (192.168.1.2) 18.06ms
          2017-05-26 08:43:37,833 [tornado.access                                     ][INFO    ][103027] 200 POST / (192.168.1.2) 10020.56ms
          2017-05-26 08:43:37,834 [tornado.application                                ][ERROR   ][103027] Future exception was never retrieved: TimeoutException
          

          To get this error, just run cmd.run salt with "sleep 500" parameter for example.

          case two

          Accidental yaml format problem. Bad pillar:

          include:
            - shared.team1:
                key: team
            pokazimTo <- this will not like yaml parser ;)
          
          environment:
            name: accGrp
          

          result from saltOutput.json:

          [{"acc3.example.sk": "The minion function caused an exception: Traceback (most recent call last):\n  File \"/usr/lib/python2.6/site-packages/salt/minion.py\", line 1445, in _thread_return\n    return_data = executor.execute()\n  File \"/usr/lib/python2.6/site-packages/salt/executors/direct_call.py\", line 28, in execute\n    return self.func(*self.args, **self.kwargs)\n  File \"/usr/lib/python2.6/site-packages/salt/modules/state.py\", line 815, in highstate\n    err += __pillar__['_errors']\n  File \"/usr/lib/python2.6/site-packages/salt/utils/context.py\", line 211, in __getitem__\n    return self._dict()[key]\nKeyError: '_errors'\n","acc1.example.sk": ["Pillar failed to render with the following messages:","Rendering SLS 'backend.acc' failed. Please see master log for details."]}]
          
          Show
          lubo_varga_nike Lubo Varga added a comment - - edited case one error case is, when in output file is saved this:  [{}] But on salt server in file /var/log/salt/api I see timeout errors: 2017-05-26 08:43:27,811 [tornado.access ][INFO ][103027] 200 POST /login (192.168.1.2) 18.06ms 2017-05-26 08:43:37,833 [tornado.access ][INFO ][103027] 200 POST / (192.168.1.2) 10020.56ms 2017-05-26 08:43:37,834 [tornado.application ][ERROR ][103027] Future exception was never retrieved: TimeoutException To get this error, just run cmd.run salt with "sleep 500" parameter for example. case two Accidental yaml format problem. Bad pillar: include: - shared.team1: key: team pokazimTo <- this will not like yaml parser ;) environment: name: accGrp result from saltOutput.json: [{ "acc3.example.sk" : "The minion function caused an exception: Traceback (most recent call last):\n File \" /usr/lib/python2.6/site-packages/salt/minion.py\ ", line 1445, in _thread_return\n return_data = executor.execute()\n File \" /usr/lib/python2.6/site-packages/salt/executors/direct_call.py\ ", line 28, in execute\n return self.func(*self.args, **self.kwargs)\n File \" /usr/lib/python2.6/site-packages/salt/modules/state.py\ ", line 815, in highstate\n err += __pillar__[ '_errors' ]\n File \" /usr/lib/python2.6/site-packages/salt/utils/context.py\ ", line 211, in __getitem__\n return self._dict()[key]\nKeyError: '_errors' \n" , "acc1.example.sk" : [ "Pillar failed to render with the following messages:" , "Rendering SLS 'backend.acc' failed. Please see master log for details." ]}]
          Hide
          mchugh19 Christian McHugh added a comment -

          Thanks for those updates!

          Case 1 is related to jenkins plugin sending a interrupt to jobs lasting more than 5 minutes. This will be handled by refactoring the plugin to run in a separate thread. This activity is being tracked in https://github.com/jenkinsci/saltstack-plugin/issues/87

          Case 2 is now handled as of https://github.com/jenkinsci/saltstack-plugin/pull/93 The test cases were added and validation corrected. This will be included in the next plugin version.

          Thanks again!

          Show
          mchugh19 Christian McHugh added a comment - Thanks for those updates! Case 1 is related to jenkins plugin sending a interrupt to jobs lasting more than 5 minutes. This will be handled by refactoring the plugin to run in a separate thread. This activity is being tracked in https://github.com/jenkinsci/saltstack-plugin/issues/87 Case 2 is now handled as of https://github.com/jenkinsci/saltstack-plugin/pull/93 The test cases were added and validation corrected. This will be included in the next plugin version. Thanks again!
          Hide
          mchugh19 Christian McHugh added a comment -

          Fixes for both your case 1 and case 2 were released in version 3.1.0. Thus I am closing this issue, but feel free to reopen if you have any problems.

          Show
          mchugh19 Christian McHugh added a comment - Fixes for both your case 1 and case 2 were released in version 3.1.0. Thus I am closing this issue, but feel free to reopen if you have any problems.
          Hide
          mchugh19 Christian McHugh added a comment -

          Fixed in release 3.1.0

          Show
          mchugh19 Christian McHugh added a comment - Fixed in release 3.1.0

            People

            Assignee:
            mchugh19 Christian McHugh
            Reporter:
            lubo_varga_nike Lubo Varga
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: