combination of java6 u23 and maven 3.0.3 made it work. So suppose this can be closed.
Still cant edit the PAMSecurityReam.java to "extends AbstractPasswordBasedSecurityRealm", i get an error:

[ERROR] Failed to execute goal org.kohsuke.stapler:maven-stapler-plugin:1.15:apt-compile (default-apt-compile) on project jenkins-core: Compilation failure
[ERROR] /home/ir/jenkins/core/src/main/java/hudson/security/PAMSecurityRealm.java:[69,7] hudson.security.PAMSecurityRealm is not abstract and does not override abstract method loadUserByUsername(java.lang.String) in hudson.security.AbstractPasswordBasedSecurityRealm
[ERROR] -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.kohsuke.stapler:maven-stapler-plugin:1.15:apt-compile (default-apt-compile) on project jenkins-core: Compilation failure
/home/ir/jenkins/core/src/main/java/hudson/security/PAMSecurityRealm.java:[69,7] hudson.security.PAMSecurityRealm is not abstract and does not override abstract method loadUserByUsername(java.lang.String) in hudson.security.AbstractPasswordBasedSecurityRealm

at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:213)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:319)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
Caused by: org.apache.maven.plugin.CompilationFailureException: Compilation failure
/home/ir/jenkins/core/src/main/java/hudson/security/PAMSecurityRealm.java:[69,7] hudson.security.PAMSecurityRealm is not abstract and does not override abstract method loadUserByUsername(java.lang.String) in hudson.security.AbstractPasswordBasedSecurityRealm

at org.kohsuke.stapler.AbstractCompilerMojo.execute(AbstractCompilerMojo.java:508)
at org.kohsuke.stapler.CompilerMojo.execute(CompilerMojo.java:111)
at org.kohsuke.stapler.AptMojo.execute(AptMojo.java:21)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
... 19 more

The error in your comment indicates that your change is incomplete. Yes, I agree that PAMSecurityRealm should extend AbstractPasswordBasedSecurityRealm. When you get your changes work, please open a pull request so that we can merge your change.

Hmmm. I have to create the abstract methods to override the ones in AbstractPasswordBasedSecurityRealm.java? but Im not a proper developer. Has someone else not already done this for PAMSecurityRealm like you have for the LDAP realm?

I have extended the Pam realm class to extend AbstractPasswordBasedSecurityRealm and have added in the abstract methods from LDAP realm, but i cant get the createSecurityComponents bits to work.

logging in via the web interface works after compiling (without errors) but the cli doesnt work. It accepts username and pass args but bombs out with...

====

java.lang.ClassCastException: java.lang.String cannot be cast to org.acegisecurity.userdetails.UserDetails
at hudson.security.PAMSecurityRealm.authenticate(PAMSecurityRealm.java:137)
at hudson.security.AbstractPasswordBasedSecurityRealm$1.authenticate(AbstractPasswordBasedSecurityRealm.java:81)
at hudson.cli.CLICommand.main(CLICommand.java:178)
at hudson.cli.CliManagerImpl.main(CliManagerImpl.java:82)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:274)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:255)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:215)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:287)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
=====

It's nearly there, but like i say, i don't do enough programming to make it work

I can post the PAMSecurityRealm.java i have thus far?

Added pull request with changes to PAMSecurityRealm

In the current head (1.421-SNAPSHOT) the PAMSecurityRealm file has been changed as per pull request but this PAM auth doesnt work (browser or cli) at all after I build jenkins.

exception:
javax.servlet.ServletException: PWC1243: Filter execution threw an exception

root cause:
java.lang.NoClassDefFoundError: Could not initialize class org.jvnet.libpam.impl.PAMLibrary$pam_conv

From the glassfish logs i see:
ThreadName=Thread-1;|StandardWrapperValve[Stapler]: PWC1406: Servlet.service() for servlet Stapler threw exception
java.lang.NoClassDefFoundError: Could not initialize class org.jvnet.libpam.impl.PAMLibrary$pam_conv
at org.jvnet.libpam.PAM.<init>(PAM.java:73)
at hudson.security.PAMSecurityRealm.authenticate(PAMSecurityRealm.java:82)
at hudson.security.AbstractPasswordBasedSecurityRealm$Authenticator.retrieveUser(AbstractPasswordBasedSecurityRealm.java:137)
at org.acegisecurity.providers.dao.AbstractUserDetailsAuthenticationProvider.authenticate(AbstractUserDetailsAuthenticationProv
ider.java:119)
at org.acegisecurity.providers.ProviderManager.doAuthentication(ProviderManager.java:195)
at org.acegisecurity.AbstractAuthenticationManager.authenticate(AbstractAuthenticationManager.java:45)
at org.acegisecurity.ui.webapp.AuthenticationProcessingFilter.attemptAuthentication(AuthenticationProcessingFilter.java:71)
at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:252)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:173)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:215)
at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:215)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:279)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:655)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:595)
at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:98)
at com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:91)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:162)
at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:326)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:227)

I have build the latest jenkins (v1.422-SNAPSHOT) from gitub with your changes for PAMSecurity realm on debian squeeze. This works now for web app and jenkins cli. Thanks

As per last comment, this seems to be fixed