Uploaded image for project: 'Infrastructure'
  1. Infrastructure
  2. INFRA-3037

idempotent-cli for our Jenkins controllers is not working for restart and reload because missing admin rights

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Puppet agent on ci.jenkins.io, trusted.ci and cert.ci reports the following errors when a configuration as code file is defined in hieradata and changed (or created):

      puppet-agent[16289]: (/Stage[main]/Profile::Buildmaster/Exec[perform-jcasc-reload]) Failed to call refresh: '/usr/share/jenkins/idempotent-cli reload-jcasc-configuration' returned 255 instead of one of [0]
      puppet-agent[16289]: (/Stage[main]/Profile::Buildmaster/Exec[perform-jcasc-reload]) '/usr/share/jenkins/idempotent-cli reload-jcasc-configuration' returned 255 instead of one of [0]
      

      which requires us to manually reload the configuration through the UI (which is not really good given the goal is to automate...)

      Thanks to Daniel Beck , we now know that it's because the CLI must be authenticated with a Jenkins account that has the administer rights as per https://github.com/jenkinsci/configuration-as-code-plugin/blob/e642ad5580416744ae761c31800c1b848d73bc53/plugin/src/main/java/io/jenkins/plugins/casc/cli/ReloadJCascConfigurationCommand.java#L26

      To the question "but WHY does the test does not catch this?", the answers are:

      • Unit test are not aimed at catching such an error
      • Vagrant integration tests/manual validation have to be improved to be closer to production:
        • Missing front apache (with local name and self signed cert)
        • No authentication setup for Jenkins in vagrant

      As a reminder, the CLI `idemptoent-cli` is a shell script on the VM which executes commands within the Jenkins container with `docker exec`: https://github.com/jenkins-infra/jenkins-infra/blob/production/dist/profile/files/buildmaster/idempotent-cli

      It has 3 uses case as far as I know:

      1. safe-restart https://github.com/jenkins-infra/jenkins-infra/blob/production/dist/profile/manifests/buildmaster.pp#L365 when a plugin has been installed or upgraded

      2. https://github.com/jenkins-infra/jenkins-infra/blob/production/dist/profile/manifests/buildmaster.pp#L374 when a change has been done on the Casc config

      3. when a plugin is installed https://github.com/jenkins-infra/jenkins-infra/blob/production/dist/profile/manifests/jenkinsplugin.pp#L15

      The case 3. (eg. installing a plugin) is calling the latest `jenkins-plugin-cli` present in the Docker LTS image of Jenkins (e.g. `docker exec` is required)

      The cases 1. and 2. both require an admin credential on jenkins.
      Running outside the container (e.g. avoid using `docker exec`) might not be required:

      • It would require adapting the script to retrieve the jenkins CLI and that Java is present outside the container
      • But it would avoid exposing the credential inside the container (if we suffer from a contaienr escape, then we would have other issues and the admin access to Jenkins would be granted anyway)

        Attachments

          Activity

          Show
          dduportal Damien Duportal added a comment - https://github.com/jenkins-infra/jenkins-infra/pull/1830
          Hide
          dduportal Damien Duportal added a comment -

          Merged and deployed

          Show
          dduportal Damien Duportal added a comment - Merged and deployed
          Hide
          dduportal Damien Duportal added a comment - - edited
          • SSHD server disabled on ci.jenkins.io
          • Removd the script create-user-cli
          • Removed the user "jenkins" in Jenkins (first with the UI, then on the file system)
          • Remove the coupld of key "jenkins-cli-key" from "/var/lib/jenkins/.ssh"
          Show
          dduportal Damien Duportal added a comment - - edited SSHD server disabled on ci.jenkins.io Removd the script create-user-cli Removed the user "jenkins" in Jenkins (first with the UI, then on the file system) Remove the coupld of key "jenkins-cli-key" from "/var/lib/jenkins/.ssh"

            People

            Assignee:
            dduportal Damien Duportal
            Reporter:
            dduportal Damien Duportal
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: