Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-64959

Auth fail on provisioned agent

    XMLWordPrintable

    Details

    • Type: New Feature
    • Status: Closed (View Workflow)
    • Priority: Critical
    • Resolution: Duplicate
    • Component/s: _unsorted
    • Labels:
      None
    • Environment:
    • Similar Issues:

      Description

      I created a new managed disk azure image with sshd_config as follows:

      ```

      PermitRootLogin without-password

      AuthorizedKeysCommand /usr/local/bin/userkeys.sh

      AuthorizedKeysCommandUser nobody

      AuthorizedKeysFile .ssh/authorized_keys

      ChallengeResponseAuthentication no

      UsePAM yes

      X11Forwarding yes

      PrintMotd no

      AcceptEnv LANG LC_*

      Subsystem sftp /usr/lib/openssh/sftp-server

      ```

      VM has user names `jenkins` with ssh keys to allow ssh key auth from Jenkins. Tested and working. 

       

      When a template uses this image the VM is created and Auth fails

      ```

       

      {{2021-02-24 16:52:34.877+0000 [id=464371] SEVERE c.m.a.v.r.AzureVMAgentSSHLauncher#getRemoteSession: AzureVMAgentSSHLauncher: getRemoteSession: Got exception while connecting to remote host linux-image-test369e60.westeurope.cloudapp.azure.com:22
      com.jcraft.jsch.JSchException: Auth fail
      at com.jcraft.jsch.Session.connect(Session.java:512)
      at com.jcraft.jsch.Session.connect(Session.java:183)
      at com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher.getRemoteSession(AzureVMAgentSSHLauncher.java:307)
      at com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher.connectToSsh(AzureVMAgentSSHLauncher.java:465)
      at com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher.launch(AzureVMAgentSSHLauncher.java:115)
      at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:294)
      at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
      at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)}}

      ```

       

      Other images created in the same manner works as expected. 

       

      Debug of SSHD on VM which fails 

      ```

      Feb 25 13:30:34 linux-image-test907690 sshd[2348]: debug1: Forked child 2399.
      Feb 25 13:30:34 linux-image-test907690 sshd[2399]: debug1: Set /proc/self/oom_score_adj to 0
      Feb 25 13:30:34 linux-image-test907690 sshd[2399]: debug1: rexec start in 5 out 5 newsock 5 pipe 7 sock 8
      Feb 25 13:30:34 linux-image-test907690 sshd[2399]: debug1: inetd sockets after dupping: 3, 3
      Feb 25 13:30:34 linux-image-test907690 sshd[2399]: Connection from 10.0.0.100 port 40088 on 10.43.240.4 port 22
      Feb 25 13:30:34 linux-image-test907690 sshd[2399]: debug1: Client protocol version 2.0; client software version JSCH-0.1.53
      Feb 25 13:30:34 linux-image-test907690 sshd[2399]: debug1: no match: JSCH-0.1.53
      Feb 25 13:30:34 linux-image-test907690 sshd[2399]: debug1: Local version string SSH-2.0-OpenSSH_7.9p1 Debian-10+deb10u2
      Feb 25 13:30:34 linux-image-test907690 sshd[2399]: debug1: permanently_set_uid: 107/65534 [preauth]
      Feb 25 13:30:34 linux-image-test907690 sshd[2399]: debug1: list_hostkey_types: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
      Feb 25 13:30:34 linux-image-test907690 sshd[2399]: debug1: SSH2_MSG_KEXINIT sent [preauth]
      Feb 25 13:30:34 linux-image-test907690 sshd[2399]: debug1: SSH2_MSG_KEXINIT received [preauth]
      Feb 25 13:30:34 linux-image-test907690 sshd[2399]: debug1: kex: algorithm: ecdh-sha2-nistp256 [preauth]
      Feb 25 13:30:34 linux-image-test907690 sshd[2399]: debug1: kex: host key algorithm: ssh-rsa [preauth]
      Feb 25 13:30:34 linux-image-test907690 sshd[2399]: debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha1 compression: none [preauth]
      Feb 25 13:30:34 linux-image-test907690 sshd[2399]: debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha1 compression: none [preauth]
      Feb 25 13:30:34 linux-image-test907690 sshd[2399]: debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth]
      Feb 25 13:30:34 linux-image-test907690 sshd[2399]: debug1: rekey after 4294967296 blocks [preauth]
      Feb 25 13:30:34 linux-image-test907690 sshd[2399]: debug1: SSH2_MSG_NEWKEYS sent [preauth]
      Feb 25 13:30:34 linux-image-test907690 sshd[2399]: debug1: expecting SSH2_MSG_NEWKEYS [preauth]
      Feb 25 13:30:34 linux-image-test907690 sshd[2399]: debug1: SSH2_MSG_NEWKEYS received [preauth]
      Feb 25 13:30:34 linux-image-test907690 sshd[2399]: debug1: rekey after 4294967296 blocks [preauth]
      Feb 25 13:30:34 linux-image-test907690 sshd[2399]: debug1: KEX done [preauth]
      Feb 25 13:30:34 linux-image-test907690 sshd[2399]: debug1: userauth-request for user jenkins service ssh-connection method none [preauth]
      Feb 25 13:30:34 linux-image-test907690 sshd[2399]: debug1: attempt 0 failures 0 [preauth]
      Feb 25 13:30:34 linux-image-test907690 sshd[2399]: debug1: PAM: initializing for "jenkins"
      Feb 25 13:30:34 linux-image-test907690 sshd[2399]: debug1: PAM: setting PAM_RHOST to "10.0.0.100"
      Feb 25 13:30:34 linux-image-test907690 sshd[2399]: debug1: PAM: setting PAM_TTY to "ssh"
      Feb 25 13:30:34 linux-image-test907690 sshd[2399]: debug1: userauth-request for user jenkins service ssh-connection method password [preauth]
      Feb 25 13:30:34 linux-image-test907690 sshd[2399]: debug1: attempt 1 failures 0 [preauth]
      Feb 25 13:30:34 linux-image-test907690 sshd[2399]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.0.0.100 user=jenkins
      Feb 25 13:30:36 linux-image-test907690 sshd[2399]: debug1: PAM: password authentication failed for jenkins: Authentication failure
      Feb 25 13:30:36 linux-image-test907690 sshd[2399]: Failed password for jenkins from 10.0.0.100 port 40088 ssh2
      Feb 25 13:30:37 linux-image-test907690 sshd[2399]: error: Received disconnect from 10.0.0.100 port 40088:3: com.jcraft.jsch.JSchException: Auth fail [preauth]
      Feb 25 13:30:37 linux-image-test907690 sshd[2399]: Disconnected from authenticating user jenkins 10.0.0.100 port 40088 [preauth]
      Feb 25 13:30:37 linux-image-test907690 sshd[2399]: debug1: do_cleanup [preauth]
      Feb 25 13:30:37 linux-image-test907690 sshd[2399]: debug1: monitor_read_log: child log fd closed
      Feb 25 13:30:37 linux-image-test907690 sshd[2399]: debug1: do_cleanup
      Feb 25 13:30:37 linux-image-test907690 sshd[2399]: debug1: PAM: cleanup
      Feb 25 13:30:37 linux-image-test907690 sshd[2399]: debug1: Killing privsep child 2400
      Feb 25 13:30:37 linux-image-test907690 sshd[2399]: debug1: audit_event: unhandled event 12

      ```

       

       

        Attachments

          Activity

          Hide
          hentis Henti Smith added a comment -

          Looking on VM from a working image I can see the `jenkins` user passwordis reset and then successfully logged in by the plugin to run `init.sh`

          ```

          Feb 25 11:24:26 localhost systemd-logind[414]: Watching system buttons on /dev/input/event1 (Power Button)
          Feb 25 11:24:26 localhost systemd-logind[414]: Watching system buttons on /dev/input/event2 (AT Translated Set 2 keyboard)
          Feb 25 11:24:26 localhost systemd-logind[414]: Watching system buttons on /dev/input/event0 (AT Translated Set 2 keyboard)
          Feb 25 11:24:26 localhost systemd-logind[414]: New seat seat0.
          Feb 25 11:24:34 localhost usermod[1916]: change user 'jenkins' password
          Feb 25 11:24:36 localhost sshd[2421]: Server listening on 0.0.0.0 port 22.
          Feb 25 11:24:36 localhost sshd[2421]: Server listening on :: port 22.
          Feb 25 11:25:01 localhost CRON[2643]: pam_unix(cron:session): session opened for user root by (uid=0)
          Feb 25 11:25:01 localhost CRON[2643]: pam_unix(cron:session): session closed for user root
          Feb 25 11:25:08 localhost sshd[2648]: Accepted keyboard-interactive/pam for jenkins from 62.253.227.100 port 58657 ssh2
          Feb 25 11:25:08 localhost sshd[2648]: pam_unix(sshd:session): session opened for user jenkins by (uid=0)
          Feb 25 11:25:08 localhost systemd-logind[414]: New session 2 of user jenkins.
          Feb 25 11:25:08 localhost systemd: pam_unix(systemd-user:session): session opened for user jenkins by (uid=0)
          Feb 25 11:25:19 localhost sudo: jenkins : TTY=unknown ; PWD=/home/jenkins ; USER=root ; COMMAND=/usr/bin/sh init.sh
          Feb 25 11:25:19 localhost sudo: pam_unix(sudo:session): session opened for user root by (uid=0)

          ```

          This seems to indicate that the Agent uses user/password for authentication ?

          Show
          hentis Henti Smith added a comment - Looking on VM from a working image I can see the `jenkins` user passwordis reset and then successfully logged in by the plugin to run `init.sh` ``` Feb 25 11:24:26 localhost systemd-logind [414] : Watching system buttons on /dev/input/event1 (Power Button) Feb 25 11:24:26 localhost systemd-logind [414] : Watching system buttons on /dev/input/event2 (AT Translated Set 2 keyboard) Feb 25 11:24:26 localhost systemd-logind [414] : Watching system buttons on /dev/input/event0 (AT Translated Set 2 keyboard) Feb 25 11:24:26 localhost systemd-logind [414] : New seat seat0. Feb 25 11:24:34 localhost usermod [1916] : change user 'jenkins' password Feb 25 11:24:36 localhost sshd [2421] : Server listening on 0.0.0.0 port 22. Feb 25 11:24:36 localhost sshd [2421] : Server listening on :: port 22. Feb 25 11:25:01 localhost CRON [2643] : pam_unix(cron:session): session opened for user root by (uid=0) Feb 25 11:25:01 localhost CRON [2643] : pam_unix(cron:session): session closed for user root Feb 25 11:25:08 localhost sshd [2648] : Accepted keyboard-interactive/pam for jenkins from 62.253.227.100 port 58657 ssh2 Feb 25 11:25:08 localhost sshd [2648] : pam_unix(sshd:session): session opened for user jenkins by (uid=0) Feb 25 11:25:08 localhost systemd-logind [414] : New session 2 of user jenkins. Feb 25 11:25:08 localhost systemd: pam_unix(systemd-user:session): session opened for user jenkins by (uid=0) Feb 25 11:25:19 localhost sudo: jenkins : TTY=unknown ; PWD=/home/jenkins ; USER=root ; COMMAND=/usr/bin/sh init.sh Feb 25 11:25:19 localhost sudo: pam_unix(sudo:session): session opened for user root by (uid=0) ``` This seems to indicate that the Agent uses user/password for authentication ?
          Hide
          hentis Henti Smith added a comment -

          Looking on VM from broken image there is no `jenkins` user password reset. 

          ```

          Feb 25 15:12:05 linux-image-testf43ca0 systemd-logind[689]: Watching system buttons on /dev/input/event1 (Power Button)
          Feb 25 15:12:05 linux-image-testf43ca0 systemd-logind[689]: Watching system buttons on /dev/input/event2 (AT Translated Set 2 keyboard)
          Feb 25 15:12:05 linux-image-testf43ca0 systemd-logind[689]: Watching system buttons on /dev/input/event0 (AT Translated Set 2 keyboard)
          Feb 25 15:12:05 linux-image-testf43ca0 systemd-logind[689]: New seat seat0.
          Feb 25 15:12:08 linux-image-testf43ca0 sshd[1182]: Server listening on 0.0.0.0 port 22.
          Feb 25 15:12:08 linux-image-testf43ca0 sshd[1182]: Server listening on :: port 22.
          Feb 25 15:12:11 linux-image-testf43ca0 sshd[1182]: Received signal 15; terminating.
          Feb 25 15:12:11 linux-image-testf43ca0 sshd[2158]: Server listening on 0.0.0.0 port 22.
          Feb 25 15:12:11 linux-image-testf43ca0 sshd[2158]: Server listening on :: port 22.
          Feb 25 15:12:23 linux-image-testf43ca0 sshd[2266]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.0.0.100 user=jenkins
          Feb 25 15:12:25 linux-image-testf43ca0 sshd[2266]: Failed password for jenkins from 10.0.0.100 port 34230 ssh2
          Feb 25 15:12:25 linux-image-testf43ca0 sshd[2266]: error: Received disconnect from 10.0.0.100 port 34230:3: com.jcraft.jsch.JSchException: Auth fail [preauth]
          Feb 25 15:12:25 linux-image-testf43ca0 sshd[2266]: Disconnected from authenticating user jenkins 10.0.0.100 port 34230 [preauth]

          ```

          I also noticed that the az vm user update command to reset the user password makes changes to both sshd_config and sudoers. 

           

          Does the Plugin reset the password on creation to allow the Jenkins instance to connect to the agent ?

          Show
          hentis Henti Smith added a comment - Looking on VM from broken image there is no `jenkins` user password reset.  ``` Feb 25 15:12:05 linux-image-testf43ca0 systemd-logind [689] : Watching system buttons on /dev/input/event1 (Power Button) Feb 25 15:12:05 linux-image-testf43ca0 systemd-logind [689] : Watching system buttons on /dev/input/event2 (AT Translated Set 2 keyboard) Feb 25 15:12:05 linux-image-testf43ca0 systemd-logind [689] : Watching system buttons on /dev/input/event0 (AT Translated Set 2 keyboard) Feb 25 15:12:05 linux-image-testf43ca0 systemd-logind [689] : New seat seat0. Feb 25 15:12:08 linux-image-testf43ca0 sshd [1182] : Server listening on 0.0.0.0 port 22. Feb 25 15:12:08 linux-image-testf43ca0 sshd [1182] : Server listening on :: port 22. Feb 25 15:12:11 linux-image-testf43ca0 sshd [1182] : Received signal 15; terminating. Feb 25 15:12:11 linux-image-testf43ca0 sshd [2158] : Server listening on 0.0.0.0 port 22. Feb 25 15:12:11 linux-image-testf43ca0 sshd [2158] : Server listening on :: port 22. Feb 25 15:12:23 linux-image-testf43ca0 sshd [2266] : pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.0.0.100 user=jenkins Feb 25 15:12:25 linux-image-testf43ca0 sshd [2266] : Failed password for jenkins from 10.0.0.100 port 34230 ssh2 Feb 25 15:12:25 linux-image-testf43ca0 sshd [2266] : error: Received disconnect from 10.0.0.100 port 34230:3: com.jcraft.jsch.JSchException: Auth fail [preauth] Feb 25 15:12:25 linux-image-testf43ca0 sshd [2266] : Disconnected from authenticating user jenkins 10.0.0.100 port 34230 [preauth] ``` I also noticed that the az vm user update command to reset the user password makes changes to both sshd_config and sudoers.    Does the Plugin reset the password on creation to allow the Jenkins instance to connect to the agent ?
          Hide
          hentis Henti Smith added a comment -

          Annoyingly, duplicating the working image to the test label works perfectly. I really am very confused. 

          Show
          hentis Henti Smith added a comment - Annoyingly, duplicating the working image to the test label works perfectly. I really am very confused. 
          Hide
          timja Tim Jacomb added a comment -

          All issues have been transferred to GitHub.

          See https://github.com/jenkinsci/azure-vm-agents-plugin/issues

          Search the issue title to find it.

          (This is a bulk comment and can't link to the specific issue)

          Show
          timja Tim Jacomb added a comment - All issues have been transferred to GitHub. See https://github.com/jenkinsci/azure-vm-agents-plugin/issues Search the issue title to find it. (This is a bulk comment and can't link to the specific issue)

            People

            Assignee:
            azure_devops Azure DevOps
            Reporter:
            hentis Henti Smith
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: