-
New Feature
-
Resolution: Unresolved
-
Major
-
None
In my company's environment, we have an extremely active Conjur cluster that handles millions of requests per day. We have a low, but non-zero amount of Conjur request failures, approximately 60 per day.
These failed requests are mostly due to bugs in Conjur that can't be trivially fixed by us.
In the Conjur credentials plugin, if any requests to Conjur fails, the build step fails and the entire build is usually aborted. These failed builds require a user to manually investigate and re-run the build.
We would like to have an option to enable retrying failed requests to be added to the Conjur credentials plugin. This could also just be made the default strategy, as it would help promote more resilient builds. This should ideally be done using an exponential back off strategy.
Example output from the plugin we observe due to Conjur failing to respond in time:
19:33:54 FATAL: Read timed out 19:33:54 java.net.SocketTimeoutException: Read timed out 19:33:54 at java.base/java.net.SocketInputStream.socketRead0(Native Method) 19:33:54 at java.base/java.net.SocketInputStream.socketRead(SocketInputStream.java:115) 19:33:54 at java.base/java.net.SocketInputStream.read(SocketInputStream.java:168) 19:33:54 at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140) 19:33:54 at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:478) 19:33:54 at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:472) 19:33:54 at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:160) 19:33:54 at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:111) 19:33:54 at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1506) 19:33:54 at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1416) 19:33:54 at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:456) 19:33:54 at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:427) 19:33:54 at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:318) 19:33:54 at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:282) 19:33:54 at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:167) 19:33:54 at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:257) 19:33:54 at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135) 19:33:54 at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114) 19:33:54 at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42) 19:33:54 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) 19:33:54 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) 19:33:54 at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) 19:33:54 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) 19:33:54 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) 19:33:54 at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) 19:33:54 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) 19:33:54 at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:126) 19:33:54 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) 19:33:54 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) 19:33:54 at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200) 19:33:54 at okhttp3.RealCall.execute(RealCall.java:77) 19:33:54 at org.conjur.jenkins.api.ConjurAPI.getAuthorizationToken(ConjurAPI.java:106) 19:33:54 at org.conjur.jenkins.conjursecrets.ConjurSecretCredentialsImpl.getSecret(ConjurSecretCredentialsImpl.java:79) 19:33:54 Caused: org.conjur.jenkins.exceptions.InvalidConjurSecretException: Read timed out 19:33:54 at org.conjur.jenkins.conjursecrets.ConjurSecretCredentialsImpl.getSecret(ConjurSecretCredentialsImpl.java:85) 19:33:54 at org.conjur.jenkins.conjursecrets.ConjurSecretCredentialsBinding.bind(ConjurSecretCredentialsBinding.java:65) 19:33:54 at org.jenkinsci.plugins.credentialsbinding.impl.SecretBuildWrapper.setUp(SecretBuildWrapper.java:94) 19:33:54 at hudson.model.Build$BuildExecution.doRun(Build.java:158) 19:33:54 at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:524) 19:33:54 at hudson.model.Run.execute(Run.java:1897) 19:33:54 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44) 19:33:54 at hudson.model.ResourceController.execute(ResourceController.java:101) 19:33:54 at hudson.model.Executor.run(Executor.java:442)
