-
Bug
-
Resolution: Cannot Reproduce
-
Major
-
None
-
Version 1.20.4, Jenkins 2.150.3
kubernetes, POD1: Jenkins, POD2:grafana with influxdb
Randomly the plugin bombs out and blocks any further access to the influx db. Influx pod needs to be restarted.
The build log never complains and always shows a successful message:
03:30:45 [InfluxDB Plugin] Collecting data for publication in InfluxDB... 03:30:45 [InfluxDB Plugin] Publishing data to: [url=http://localhost:8086, description=jenkins, username=admin, password=*****, database=jenkins] [Pipeline] } [Pipeline] // timestamps [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline java.net.SocketException: Socket closed at java.net.SocketInputStream.read(SocketInputStream.java:204) at java.net.SocketInputStream.read(SocketInputStream.java:141) at okio.Okio$2.read(Okio.java:140) at okio.AsyncTimeout$2.read(AsyncTimeout.java:237) Caused: java.net.SocketTimeoutException: timeout at okio.Okio$4.newTimeoutException(Okio.java:232) at okio.AsyncTimeout.exit(AsyncTimeout.java:285) at okio.AsyncTimeout$2.read(AsyncTimeout.java:241) at okio.RealBufferedSource.indexOf(RealBufferedSource.java:355) at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:227) at okhttp3.internal.http1.Http1Codec.readHeaderLine(Http1Codec.java:215) at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189) at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:88) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:126) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at org.influxdb.impl.BasicAuthInterceptor.intercept(BasicAuthInterceptor.java:22) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at org.influxdb.impl.GzipRequestInterceptor.intercept(GzipRequestInterceptor.java:42) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at okhttp3.logging.HttpLoggingInterceptor.intercept(HttpLoggingInterceptor.java:144) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200) at okhttp3.RealCall.execute(RealCall.java:77) at retrofit2.OkHttpCall.execute(OkHttpCall.java:180) at org.influxdb.impl.InfluxDBImpl.execute(InfluxDBImpl.java:788) Caused: org.influxdb.InfluxDBIOException at org.influxdb.impl.InfluxDBImpl.execute(InfluxDBImpl.java:800) at org.influxdb.impl.InfluxDBImpl.write(InfluxDBImpl.java:455) at jenkinsci.plugins.influxdb.InfluxDbPublicationService.writeToInflux(InfluxDbPublicationService.java:299) Caused: jenkinsci.plugins.influxdb.InfluxReportException at jenkinsci.plugins.influxdb.InfluxDbPublicationService.writeToInflux(InfluxDbPublicationService.java:302) at jenkinsci.plugins.influxdb.InfluxDbPublicationService.perform(InfluxDbPublicationService.java:273) at jenkinsci.plugins.influxdb.InfluxDbPublisher.perform(InfluxDbPublisher.java:357) at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:80) at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:67) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
The Jenkins Admin logs console you can see:
Apr 12, 2019 3:30:45 PM FINE jenkinsci.plugins.influxdb.InfluxDbPublicationService Data source empty: Custom Data Apr 12, 2019 3:30:45 PM FINE jenkinsci.plugins.influxdb.InfluxDbPublicationService Data source empty: Custom Data Map Apr 12, 2019 3:30:45 PM FINE jenkinsci.plugins.influxdb.InfluxDbPublicationService Plugin skipped: SonarQube Apr 12, 2019 3:30:45 PM FINE jenkinsci.plugins.influxdb.InfluxDbPublicationService Data source empty: Change Log Apr 12, 2019 3:30:45 PM FINE jenkinsci.plugins.influxdb.InfluxDbPublicationService Plugin skipped: Performance Publisher Apr 12, 2019 3:30:45 PM FINE jenkinsci.plugins.influxdb.InfluxDbPublicationService [InfluxDB Plugin] Publishing data to: [url=http://localhost:8086, description=jenkins, username=admin, password=*****, database=jenkins] Apr 12, 2019 3:30:55 PM WARNING jenkinsci.plugins.influxdb.InfluxDbPublicationService writeToInflux Could not report to InfluxDB. Ignoring Exception. java.net.SocketTimeoutException: timeout at okio.Okio$4.newTimeoutException(Okio.java:232) at okio.AsyncTimeout.exit(AsyncTimeout.java:275) at okio.AsyncTimeout$2.read(AsyncTimeout.java:243) at okio.RealBufferedSource.indexOf(RealBufferedSource.java:355) at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:227) at okhttp3.internal.http1.Http1Codec.readHeaderLine(Http1Codec.java:215) at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189) at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:88) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:126) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at org.influxdb.impl.BasicAuthInterceptor.intercept(BasicAuthInterceptor.java:22) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at org.influxdb.impl.GzipRequestInterceptor.intercept(GzipRequestInterceptor.java:42) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at okhttp3.logging.HttpLoggingInterceptor.intercept(HttpLoggingInterceptor.java:144) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200) at okhttp3.RealCall.execute(RealCall.java:77) at retrofit2.OkHttpCall.execute(OkHttpCall.java:180) at org.influxdb.impl.InfluxDBImpl.execute(InfluxDBImpl.java:788) Caused: org.influxdb.InfluxDBIOException at org.influxdb.impl.InfluxDBImpl.execute(InfluxDBImpl.java:800) at org.influxdb.impl.InfluxDBImpl.write(InfluxDBImpl.java:455) at jenkinsci.plugins.influxdb.InfluxDbPublicationService.writeToInflux(InfluxDbPublicationService.java:299) at jenkinsci.plugins.influxdb.InfluxDbPublicationService.perform(InfluxDbPublicationService.java:273) at jenkinsci.plugins.influxdb.InfluxDbPublisher.perform(InfluxDbPublisher.java:357) at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:80) at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:67) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
Reproducible randomly with a pipeline like:
node {
try {
stage('Clone') {
sh 'dosomething'
}
} catch (Exception e) {
if(currentBuild.result == null) {
currentBuild.result = "FAILURE"
}
} finally {
step([$class: 'InfluxDbPublisher', jenkinsEnvParameterTag: 'EXEC=' + "dummy", target: 'jenkins'])
}
}
Could you suggest a workaround or a way to increase the log level to see where it fails?
[JENKINS-56993] Influx not responsive after error in log
Fred'O
added a comment - I stopped using jenkinsEnvParameterTag and use jenkinsEnvParameterField instead which never returned any issue.
Fred'O
added a comment - I stopped using jenkinsEnvParameterTag and use jenkinsEnvParameterField instead which never returned any issue. 
Are you still facing this issue? Unfortunately I've never seen this error before. I've just used Docker on my local stuff without Kubernetes and I believe this is an error related more to Kubernetes connectivity.
It could be possible to add some sort of connection listening to the plugin to ensure we have a connection.