-
Bug
-
Resolution: Cannot Reproduce
-
Major
-
Jenkins ver. 2.73.3
P4 Plugin ver. 1.8.12
I created a very simple pipeline job to test polling without a sync (preview only mode) and it is failing with the below error:
P4 Task: failed: java.lang.NullPointerException
There is no exception logged in the Jenkins log. The exception seemed to occur on the slave, so I checked the swarm log and nothing was found.
Here is the pipeline script:
pipeline {
agent {
label "CentOS6"
}
stages {
stage("Init") {
steps {
checkout perforce(
credential: 'p4-hbase-build',
populate: previewOnly(quiet: true),
workspace: manualSpec(
charset: 'none',
name: 'jenkins-${NODE_NAME}-${JOB_NAME}-${EXECUTOR_NUMBER}',
pinHost: false,
spec: clientSpec(
allwrite: false,
backup: false,
clobber: true,
compress: false,
line: 'LOCAL',
locked: false,
modtime: false,
rmdir: false,
serverID: '',
streamName: '',
type: 'WRITABLE',
view: '//sandbox/hdara/test/... //jenkins-${NODE_NAME}-${JOB_NAME}-${EXECUTOR_NUMBER}/...'
)
)
)
}
}
stage("Run") {
steps {
echo "INFO: Running..."
sh "export"
}
}
}
}
Here is the job console output:
Started by user Hari Krishna Dara
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] node
Running on slave-a6209140 in /home/jenkins/workspace/test
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Init)
[Pipeline] checkout
... p4 client -o jenkins-slave-a6209140-test-0 +
... p4 info +
... p4 client -o jenkins-slave-a6209140-test-0 +
... p4 info +
... p4 info +
... p4 client -i +
... client: jenkins-slave-a6209140-test-0
... p4 client -o jenkins-slave-a6209140-test-0 +
... p4 info +
... p4 counter change +
... p4 changes -m1 -ssubmitted //jenkins-slave-a6209140-t___ +
Building on Node: slave-a6209140
... p4 client -o jenkins-slave-a6209140-test-0 +
... p4 info +
P4 Task: establishing connection.
... server: p4proxy-rep:1999
... node: slave
P4 Task: attempt: 1
P4 Task: failed: java.lang.NullPointerException
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Run)
Stage "Run" skipped due to earlier failure(s)
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: P4 Task: failed: java.lang.NullPointerException
Finished: FAILURE
I tried to capture the NPE by starting swarm JVM in debug mode and put a breakpoint via jdb. Here is the stacktrace:
[1] org.jenkinsci.plugins.p4.client.ClientHelper.tidyWorkspace (ClientHelper.java:449) [2] org.jenkinsci.plugins.p4.tasks.CheckoutTask.task (CheckoutTask.java:143) [3] org.jenkinsci.plugins.p4.tasks.AbstractTask.tryTask (AbstractTask.java:211) [4] org.jenkinsci.plugins.p4.tasks.CheckoutTask.invoke (CheckoutTask.java:137) [5] org.jenkinsci.plugins.p4.tasks.CheckoutTask.invoke (CheckoutTask.java:30) [6] hudson.FilePath$FileCallableWrapper.call (FilePath.java:2,750) [7] hudson.remoting.UserRequest.perform (UserRequest.java:153) [8] hudson.remoting.UserRequest.perform (UserRequest.java:50) [9] hudson.remoting.Request$2.run (Request.java:336) [10] hudson.remoting.InterceptingExecutorService$1.call (InterceptingExecutorService.java:68) [11] java.util.concurrent.FutureTask.run (FutureTask.java:266) [12] java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1,142) [13] java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:617) [14] hudson.remoting.Engine$1$1.run (Engine.java:94) [15] java.lang.Thread.run (Thread.java:745)
Looks like iclient is null as seen in the below jdb output:
pool-1-thread-11 for JNLP4-connect connection to slave/10.253.229.26:5000 id=239[1] print iclient
iclient = null
pool-1-thread-11 for JNLP4-connect connection to slave/10.253.229.26:5000 id=239[1] print connection
connection = "com.perforce.p4java.impl.mapbased.rpc.OneShotServerImpl@104446f1"
pool-1-thread-11 for JNLP4-connect connection to slave/10.253.229.26:5000 id=239[1] print connection.getClient("jenkins-slave-a6209140-test-0")
connection.getClient("jenkins-slave-a6209140-test-0") = null
pool-1-thread-11 for JNLP4-connect connection to slave/10.253.229.26:5000 id=239[1] print connection.getClass().getName()
connection.getClass().getName() = "com.perforce.p4java.impl.mapbased.rpc.OneShotServerImpl"
pool-1-thread-11 for JNLP4-connect connection to slave/10.253.229.26:5000 id=239[1] print connection.getClientName()
connection.getClientName() = null
pool-1-thread-11 for JNLP4-connect connection to slave/10.253.229.26:5000 id=239[1] print connection.getServerHostPort()
connection.getServerHostPort() = "p4proxy-rep:1999"
pool-1-thread-11 for JNLP4-connect connection to slave/10.253.229.26:5000 id=239[1] print connection.getAuthTicket("build")
connection.getAuthTicket("build") = "FG9435026878130GD728FD8E02A77DD2"
[JENKINS-52635] Pipeline job using P4 failing with NPE
Hari Dara
created issue -
Hari Dara
added a comment - Unfortunately, this is not just limited to pipeline. I tried a Freestyle project with all defaults (except for the credential and mapping) and it failed with the same NPE:
Started by an SCM change [EnvInject] - Loading node environment variables. Building remotely on slave-a6209140 (swarm CentOS6 Ambari slave) in workspace /home/jenkins/workspace/test ... p4 client -o jenkins-slave-a6209140-test-0 + ... p4 info + ... p4 client -o jenkins-slave-a6209140-test-0 + ... p4 info + ... p4 info + ... p4 client -i + ... client: jenkins-slave-a6209140-test-0 ... p4 client -o jenkins-slave-a6209140-test-0 + ... p4 info + ... p4 counter change + ... p4 changes -m1 -ssubmitted //jenkins-slave-a6209140-t___ + Building on Node: slave-a6209140 ... p4 client -o jenkins-slave-a6209140-test-0 + ... p4 info +P4 Task: establishing connection. ... server: p4proxy-rep:1999 ... node: slaveP4 Task: attempt: 1 P4 Task: failed: java.lang.NullPointerException ERROR: P4 Task: failed: java.lang.NullPointerException Finished: FAILURE
I am assuming something is different in our perforce environment that is causing the plugin to hit a bug, so any help in diagnosing this issue would be appreciated.
Hari Dara
added a comment - Unfortunately, this is not just limited to pipeline. I tried a Freestyle project with all defaults (except for the credential and mapping) and it failed with the same NPE:
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on slave-a6209140 (swarm CentOS6 Ambari slave) in workspace /home/jenkins/workspace/test
... p4 client -o jenkins-slave-a6209140-test-0 +
... p4 info +
... p4 client -o jenkins-slave-a6209140-test-0 +
... p4 info +
... p4 info +
... p4 client -i +
... client: jenkins-slave-a6209140-test-0
... p4 client -o jenkins-slave-a6209140-test-0 +
... p4 info +
... p4 counter change +
... p4 changes -m1 -ssubmitted //jenkins-slave-a6209140-t___ +
Building on Node: slave-a6209140
... p4 client -o jenkins-slave-a6209140-test-0 +
... p4 info +P4 Task: establishing connection.
... server: p4proxy-rep:1999
... node: slaveP4 Task: attempt: 1
P4 Task: failed: java.lang.NullPointerException
ERROR: P4 Task: failed: java.lang.NullPointerException
Finished: FAILURE
I am assuming something is different in our perforce environment that is causing the plugin to hit a bug, so any help in diagnosing this issue would be appreciated. Hari Dara
added a comment - I have narrowed down the issue to the p4port being used. The port p4proxy-rep:9999 that I was using is a new RO proxy that is meant for automated tools that our SCM team recently introduced. When I switched to the regular p4port that is meant for developers, the problem went away. I don't know how this RO proxy is setup, so I won't be able to provide any reproducible information yet, but I am looping them in. Meanwhile, I have a workaround and so don't consider this as a serious issue.
Hari Dara
added a comment - I have narrowed down the issue to the p4port being used. The port p4proxy-rep:9999 that I was using is a new RO proxy that is meant for automated tools that our SCM team recently introduced. When I switched to the regular p4port that is meant for developers, the problem went away. I don't know how this RO proxy is setup, so I won't be able to provide any reproducible information yet, but I am looping them in. Meanwhile, I have a workaround and so don't consider this as a serious issue. Hari Dara
made changes -
| Priority | Original: Critical [ 2 ] | New: Major [ 3 ] |
Paul Allen
added a comment - Thank you for the detailed investigation. Sorry the error reporting is not as good as it could be; I have been making improvements to raising errors back from slaves, but connection must have been overlooked. Please can you use a regular 'p4' command line client and run a 'p4 -p p4proxy-rep:9999 -ztag info' that might tell me some details about the type of server/proxy being used.
Paul Allen
added a comment - Thank you for the detailed investigation. Sorry the error reporting is not as good as it could be; I have been making improvements to raising errors back from slaves, but connection must have been overlooked. Please can you use a regular 'p4' command line client and run a 'p4 -p p4proxy-rep:9999 -ztag info' that might tell me some details about the type of server/proxy being used. Paul Allen
added a comment - Please **** out any fields in the 'p4 info' report that might be company sensitive.
Paul Allen
added a comment - Please **** out any fields in the 'p4 info' report that might be company sensitive. Hari Dara
added a comment - Thanks for responding and for the helpful warning (though I am already doing it). I am out of station now and will try to get back with the information in a couple of days.
Hari Dara
added a comment - Thanks for responding and for the helpful warning (though I am already doing it). I am out of station now and will try to get back with the information in a couple of days. Hari Dara
added a comment - Here is the output of p4 -ztag info:
... proxyVersion P4P/LINUX26X86_64/2017.1/1511680 (2017/05/05) ... proxyAddress 10.253.136.196:1999 ... proxyEncryption encrypted ... proxyCertExpires Sep 25 23:59:59 2018 GMT ... proxyRoot /***/perforce/proxyroot ... brokerAddress scm-pfourbroker2-2:1666 ... brokerEncryption encrypted ... brokerCertExpire Sep 25 23:59:59 2018 GMT ... brokerVersion P4BROKER/LINUX26X86_64/2016.2/1487173 ... userName build ... password enabled ... clientName *unknown* ... clientCwd /home/jenkins ... clientHost slave ... clientCase sensitive ... peerAddress 10.253.137.179:56886 ... clientAddress 10.253.229.10 ... serverName scm-pfourreplica3-2 ... lbr.replication readonly ... monitor enabled ... security enabled ... serverAddress scm-pfourreplica3-2 ... serverRoot /perforce/db ... serverDate 2018/07/22 02:33:00 -0700 PDT ... tzoffset -25200 ... serverUptime 2197:51:42 ... serverVersion P4D/LINUX26X86_64/2016.2/1509269 (2017/04/27) ... serverEncryption encrypted ... serverCertExpires Sep 25 23:59:59 2018 GMT ... ServerID scm-pfourreplica3-2 ... serverServices standard ... serverCluster mainAuth ... serverLicense *** ... serverLicense-ip 127.0.0.1 ... caseHandling sensitive ... replica ssl:p4edge1-rep:1666 ... integEngine 3 ... transportInfo retransmits 0 probes 0 backoff 0 options timestamps sack wscale sscale 7 rscale 7 rto 203000 ato 40000 snd_mss 1448 rcv_mss 677 unacked 0 sacked 0 lost 0 retrans 0 fackets 0 last_data_sent 1 last_data_recv 0 last_ack_recv 0 pmtu 1500 rcv_ssthresh 17978 rtt 3000 rttvar 2750 snd_ssthresh 2147483647 snd_cwnd 12 advmss 1448 reordering 18
Hari Dara
added a comment - Here is the output of p4 -ztag info :
... proxyVersion P4P/LINUX26X86_64/2017.1/1511680 (2017/05/05)
... proxyAddress 10.253.136.196:1999
... proxyEncryption encrypted
... proxyCertExpires Sep 25 23:59:59 2018 GMT
... proxyRoot /***/perforce/proxyroot
... brokerAddress scm-pfourbroker2-2:1666
... brokerEncryption encrypted
... brokerCertExpire Sep 25 23:59:59 2018 GMT
... brokerVersion P4BROKER/LINUX26X86_64/2016.2/1487173
... userName build
... password enabled
... clientName *unknown*
... clientCwd /home/jenkins
... clientHost slave
... clientCase sensitive
... peerAddress 10.253.137.179:56886
... clientAddress 10.253.229.10
... serverName scm-pfourreplica3-2
... lbr.replication readonly
... monitor enabled
... security enabled
... serverAddress scm-pfourreplica3-2
... serverRoot /perforce/db
... serverDate 2018/07/22 02:33:00 -0700 PDT
... tzoffset -25200
... serverUptime 2197:51:42
... serverVersion P4D/LINUX26X86_64/2016.2/1509269 (2017/04/27)
... serverEncryption encrypted
... serverCertExpires Sep 25 23:59:59 2018 GMT
... ServerID scm-pfourreplica3-2
... serverServices standard
... serverCluster mainAuth
... serverLicense ***
... serverLicense-ip 127.0.0.1
... caseHandling sensitive
... replica ssl:p4edge1-rep:1666
... integEngine 3
... transportInfo retransmits 0 probes 0 backoff 0
options timestamps sack wscale
sscale 7 rscale 7
rto 203000 ato 40000 snd_mss 1448 rcv_mss 677
unacked 0 sacked 0 lost 0 retrans 0 fackets 0
last_data_sent 1 last_data_recv 0 last_ack_recv 0
pmtu 1500 rcv_ssthresh 17978 rtt 3000 rttvar 2750
snd_ssthresh 2147483647 snd_cwnd 12 advmss 1448 reordering 18
Paul Allen
added a comment - Thank you for the output:
lbr.replication readonly
Your admin might have setup a legacy 'Build farm' server; for modern CI systems like Jenkins a Commit-edge server would a better alternative. Please refer to the note in:
https://www.perforce.com/perforce/doc.current/manuals/p4dist/index.html#P4Dist/DB5-72814.html
Officially the p4-plugin does not support 'Build farm' servers, however there is a limited configuration that a few customers have found to work.
Paul Allen
added a comment - Thank you for the output:
lbr.replication readonly
Your admin might have setup a legacy 'Build farm' server; for modern CI systems like Jenkins a Commit-edge server would a better alternative. Please refer to the note in:
https://www.perforce.com/perforce/doc.current/manuals/p4dist/index.html#P4Dist/DB5-72814.html
Officially the p4-plugin does not support 'Build farm' servers, however there is a limited configuration that a few customers have found to work. 
I tried using all the defaults including the populate, and I still got the same NPE. For me p4 plugin as SCM option in pipeline is completely unusable right now.