Hi,

I installed Jenkins latest stable version (2.176.3) on Ubuntu 16.04 machine. 
I chose Unix user/group database as a security realm and for authorization I chose Project-based Matrix Authorization Strategy.
Under Unix user/group database I chose the service name to be sshd, and I got "Success" by clicking the Test button.

The Jenkins is running by user named foo (NIS user) with sudo privileges and belongs to shadow group. 
I tried to connect to Jenkins web by foo user and it succeed, but when I tried to connect by a another NIS user that is not running the Jenkins service (for example: bar user) it failed and displayed "Invalid username or password" message (the machine is configured via PAM to enable NIS account to login).
The NIS user (for example: bar) can login directly to the machine, but can't login to the Jenkins web.

I checked /var/log/auth.log immediately after failure login of NIS user to Jenkins web and I saw in that file that the user successfully login, but I got the following error:
pam_krb5(sshd:setcred): (user bar) chown of ticket cache failed: Operation not permitted.
user bar is a NIS user, it can login directly to the machine, but can't login to the Jenkins web.

From my understanding, the user successfully login to the machine, and even creates a keytab under /tmp directory. But the ownership of the keytab file is foo user (Jenkins service account user). Therefore, foo user trying to change the ownership of the file to the NIS user - in this case to bar (by chown command), but only root has the privilege to change it, so it returns Operation not permitted.
When I changed the Jenkins service account user to root the NIS user (bar) succeed login to the Jenkins web.

In addition, I installed Jenkins version 2.138.4 on Ubuntu 16.04 machine and configured the security login as I configured in version 2.176.3 (Unix user/group database and Project based Matrix Authorization Strategy) and I could login by NIS user.

I found a workaround for this bug. I changed few files that belongs to PAM.
I replaced the following files by files from another Jenkins server version 2.138.4 and then the bug solved:

• replaced /var/lib/jenkins/plugins/pam-auth/WEB-INF/lib by pam-auth/WEB-INF/lib directory (from Jenkins server version 2.138.4) - it didn't help.
• replaced /var/cache/jenkins/war/WEB-INF/lib/libpam4j-1.11.jar by libpam4j-1.8.jar (from Jenkins server version 2.138.4) - helped.

More details will be provided upon in request.

Thanks,
Liran