[JENKINS-49362] Master unable to connect to GitHub using IBM JDK 8 - Unable to find acceptable protocols - Jenkins Jira

Type: Bug
Resolution: Fixed
Priority: Major
Component/s: github-api-plugin, github-branch-source-plugin, github-plugin
Labels:
- java-ibm
- triaged-2018-11
Environment:

Hide
Jenkins 2.89.3 running on IBM z13 Mainframe using IBM JDK 8
java.runtime.name Java(TM) SE Runtime Environment
java.runtime.version 8.0.5.7 - pxz6480sr5fp7-20171216_01(SR5 FP7)
java.version 1.8.0_151
java.vm.info JRE 1.8.0 Linux s390x-64 Compressed References 20171215_373586 (JIT enabled, AOT enabled)>OpenJ9 - 5aa401f>OMR - 101e793>IBM - b4a79bf

GitHub Branch Source Plugin - 2.3.2

Show
Jenkins 2.89.3 running on IBM z13 Mainframe using IBM JDK 8 java.runtime.name Java(TM) SE Runtime Environment java.runtime.version 8.0.5.7 - pxz6480sr5fp7-20171216_01(SR5 FP7) java.version 1.8.0_151 java.vm.info JRE 1.8.0 Linux s390x-64 Compressed References 20171215_373586 (JIT enabled, AOT enabled)>OpenJ9 - 5aa401f>OMR - 101e793>IBM - b4a79bf GitHub Branch Source Plugin - 2.3.2

Similar Issues:
Powered by SuggestiMate

Show
Epic Link:
Docker images for IBM s390x
Released As:
https://github.com/jenkinsci/github-branch-source-plugin/releases/tag/github-branch-source-2.8.2

Running on IBM z13 mainframe, after installing IBM JDK 8, I am unable to pull the branch source from GitHub. This is really a critical/major problem, since the only other JDK for the z13 would be OpenJDK, which uses Zero VM (interpreted mode only), so performance is terrible, so bad in fact that the machine runs at 200-300%, and cannot keeps up with long running jobs on the agents.

// code placeholderjava.net.UnknownServiceException: Unable to find acceptable protocols. isFallback=false, modes=[ConnectionSpec(cipherSuites=[TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_3DES_EDE_CBC_SHA], tlsVersions=[TLS_1_2, TLS_1_1, TLS_1_0], supportsTlsExtensions=true), ConnectionSpec(cipherSuites=[TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_3DES_EDE_CBC_SHA], tlsVersions=[TLS_1_0], supportsTlsExtensions=true), ConnectionSpec()], supported protocols=[TLSv1]
	at com.squareup.okhttp.internal.ConnectionSpecSelector.configureSecureSocket(ConnectionSpecSelector.java:73)
	at com.squareup.okhttp.internal.io.RealConnection.connectTls(RealConnection.java:185)
	at com.squareup.okhttp.internal.io.RealConnection.connectSocket(RealConnection.java:149)
	at com.squareup.okhttp.internal.io.RealConnection.connect(RealConnection.java:112)
	at com.squareup.okhttp.internal.http.StreamAllocation.findConnection(StreamAllocation.java:184)
	at com.squareup.okhttp.internal.http.StreamAllocation.findHealthyConnection(StreamAllocation.java:126)
	at com.squareup.okhttp.internal.http.StreamAllocation.newStream(StreamAllocation.java:95)
	at com.squareup.okhttp.internal.http.HttpEngine.connect(HttpEngine.java:281)
	at com.squareup.okhttp.internal.http.HttpEngine.sendRequest(HttpEngine.java:224)
	at com.squareup.okhttp.internal.huc.HttpURLConnectionImpl.execute(HttpURLConnectionImpl.java:450)
	at com.squareup.okhttp.internal.huc.HttpURLConnectionImpl.getResponse(HttpURLConnectionImpl.java:399)
	at com.squareup.okhttp.internal.huc.HttpURLConnectionImpl.getResponseCode(HttpURLConnectionImpl.java:527)
	at com.squareup.okhttp.internal.huc.DelegatingHttpsURLConnection.getResponseCode(DelegatingHttpsURLConnection.java:105)
	at com.squareup.okhttp.internal.huc.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:25)
	at org.kohsuke.github.Requester.parse(Requester.java:602)
Caused: org.kohsuke.github.HttpException: Server returned HTTP response code: -1, message: 'null' for URL: https://github.ibm.com/api/v3/
	at org.kohsuke.github.Requester.parse(Requester.java:633)
	at org.kohsuke.github.Requester.parse(Requester.java:594)
	at org.kohsuke.github.Requester._to(Requester.java:272)
	at org.kohsuke.github.Requester.to(Requester.java:234)
	at org.kohsuke.github.GitHub.checkApiUrlValidity(GitHub.java:703)
	at org.jenkinsci.plugins.github_branch_source.GitHubSCMSource.checkApiUrlValidity(GitHubSCMSource.java:1291)
Caused: java.io.IOException: It seems https://github.ibm.com/api/v3 is unreachable
	at org.jenkinsci.plugins.github_branch_source.GitHubSCMSource.checkApiUrlValidity(GitHubSCMSource.java:1294)
	at org.jenkinsci.plugins.github_branch_source.GitHubSCMSource.retrieve(GitHubSCMSource.java:1151)
	at jenkins.scm.api.SCMSource.fetch(SCMSource.java:598)
	at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrieve(SCMSourceRetriever.java:80)
	at org.jenkinsci.plugins.workflow.libs.LibraryAdder.retrieve(LibraryAdder.java:153)
	at org.jenkinsci.plugins.workflow.libs.LibraryAdder.add(LibraryAdder.java:134)
	at org.jenkinsci.plugins.workflow.libs.LibraryDecorator$1.call(LibraryDecorator.java:125)
	at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1065)
	at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603)
	at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581)
	at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558)
	at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
	at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
	at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688)
	at groovy.lang.GroovyShell.parse(GroovyShell.java:700)
	at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:129)
	at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:123)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:517)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:480)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:268)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:421)
org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
WorkflowScript: Loading libraries failed

1 error

	at org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:310)
	at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1085)
	at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603)
	at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581)
	at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558)
	at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
	at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
	at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688)
	at groovy.lang.GroovyShell.parse(GroovyShell.java:700)
	at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:129)
	at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:123)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:517)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:480)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:268)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:421)

It looks like this issue: https://github.com/watson-developer-cloud/java-sdk/issues/603 Where the a fix may have gone into okhttp.. The issue seems quite old, but it seems the problem lies in that area.

Volodymyr Paprotski added a comment - 2018-05-30 02:57 - edited

(In trying to keep 'same Java runs everywhere' conspiracy/rumours going) I spent much of the day today on this issue and am now past at least this issue. I think.

As Joe suspected, issue appears to be in okhttp. I believe this was fixed in https://github.com/square/okhttp/issues/3173 okhttp3 3.7 according to the comments. There were quite a few fixes in there, from mismatch of names (TLS_ vs SSL_) to some dangerous reflection code reaching straight into the JSSE provider internals to grab some internal secrets (IBM JCE team runs obfuscators on all their release builds, making reflection possible but impractical)

Long story short, several jenkins pluggins, that use okhttp need to update their dependencies and/or code.

What follows is a record of my 'experiment' to prove above point. (Hopefully useful to somebody)

Important note: this is my first foray into Maven, take mvn commands with grain of salt. There are likely better/correct/faster way of doing what I did.

There are three pluggins involved. I cloned and checked out the version that matches my jenkins install. First two, just update pom.xml to use okhttp3 3.8.1 (only one month newer then 3.7 in issue above). Last one involves code change.

1. git clone https://github.com/jenkinsci/github-plugin

2. git clone https://github.com/jenkinsci/github-api-plugin

3. git clone https://github.com/jenkinsci/github-branch-source-plugin

~ Verbatim

Github-plugin

(Might not be actually involved in this issue, but is using okhttp 2.7 so, updated)

$ git checkout v1.29.0

$ vim pom.xml:

<dependency>
- <groupId>com.squareup.okhttp</groupId>
+ <groupId>com.squareup.okhttp3</groupId>
{{ <artifactId>okhttp-urlconnection</artifactId>}}
- <version>2.7.5</version>
+ <version>3.8.1</version>$ mvn hpi:hpi # mkdir target/classes?

# Might I say thanks to github-plugin developers to be the only ones with a usable README.md as their front page! Saved me a lot of mvn grief

Grab target/*.hpi file and upload it to your jenkins via Manage Pluggins->Advanced

Github-api-plugin

Curiously, kohsuke of CloudBees Inc. already is using okhttp 3.9 https://github.com/kohsuke/github-api/blob/master/pom.xml#L157

$ git checkout github-api-1.90

$ vim pom.xml

{{ <groupId>org.kohsuke</groupId>}}
{{ <artifactId>github-api</artifactId>}}
- <version>1.90</version>
+ <version>1.93</version>

...{{ }}

<dependency>
- <groupId>com.squareup.okhttp</groupId>
+ <groupId>com.squareup.okhttp3</groupId>
{{ <artifactId>okhttp-urlconnection</artifactId>}}
- <version>2.7.5</version>
+ <version>3.8.1</version>

$ mvn hpi:hpi

$ mvn install -DskipTests # install locally, needed by github branch-source

Grab target/*.hpi file and upload it to your jenkins via Manage Pluggins->Advanced

Github-branch-source-plugin

$ git checkout github-branch-source-2.3.4

$ vim pom.xml # resolving build rule failure by explicitly specifying newer package at root

+ <dependency>
+ <groupId>org.apache.commons</groupId>
+ <artifactId>commons-lang3</artifactId>
+ <version>3.7</version>
+ </dependency>

$ vim src/main/java/org/jenkinsci/plugins/github_branch_source/Connector.java

-import com.squareup.okhttp.OkHttpClient;
-import com.squareup.okhttp.OkUrlFactory;
+import okhttp3.OkHttpClient;
+import okhttp3.OkUrlFactory;

...

-import org.kohsuke.github.extras.OkHttpConnector;
+import org.kohsuke.github.extras.OkHttp3Connector;

...

- OkHttpClient client = new OkHttpClient().setProxy(getProxy(host));
+ OkHttpClient client = new OkHttpClient.Builder().proxy(getProxy(host)).build();

- gb.withConnector(new OkHttpConnector(new OkUrlFactory(client)));
+ gb.withConnector(new OkHttp3Connector(new OkUrlFactory(client)));

$ mvn compile

$ mvn hpi:hpi

Grab target/*.hpi file and upload it to your jenkins via Manage Pluggins->Advanced

Now, I am able to index my Github Enterprise branches!

PS: Jenkins was started with ./ibm-java-s390x-80/bin/java -Dcom.ibm.jsse2.overrideDefaultProtocol=TLSv12 -Djavax.net.debug=ssl,handshake,data,verbose, -Djava.library.path=/usr/lib/jni -Dcom.ibm.jsse2.overrideDefaultTLS=true -jar jenkins.war

I haven't bothered removing any of the tracing and override options as of yet to see if they have any effect on the fix.

Volodymyr Paprotski added a comment - 2018-05-30 02:57 - edited (In trying to keep 'same Java runs everywhere' conspiracy/rumours going) I spent much of the day today on this issue and am now past at least this issue. I think. As Joe suspected, issue appears to be in okhttp. I believe this was fixed in https://github.com/square/okhttp/issues/3173 okhttp3 3.7 according to the comments. There were quite a few fixes in there, from mismatch of names (TLS_ vs SSL_) to some dangerous reflection code reaching straight into the JSSE provider internals to grab some internal secrets (IBM JCE team runs obfuscators on all their release builds, making reflection possible but impractical) Long story short, several jenkins pluggins, that use okhttp need to update their dependencies and/or code. What follows is a record of my 'experiment' to prove above point. (Hopefully useful to somebody) Important note: this is my first foray into Maven, take mvn commands with grain of salt. There are likely better/correct/faster way of doing what I did. There are three pluggins involved. I cloned and checked out the version that matches my jenkins install. First two, just update pom.xml to use okhttp3 3.8.1 (only one month newer then 3.7 in issue above). Last one involves code change. 1. git clone https://github.com/jenkinsci/github-plugin 2. git clone https://github.com/jenkinsci/github-api-plugin 3. git clone https://github.com/jenkinsci/github-branch-source-plugin ~ Verbatim Github-plugin (Might not be actually involved in this issue, but is using okhttp 2.7 so, updated) $ git checkout v1.29.0 $ vim pom.xml: <dependency> - <groupId>com.squareup.okhttp</groupId> + <groupId>com.squareup.okhttp3</groupId> {{ <artifactId>okhttp-urlconnection</artifactId>}} - <version>2.7.5</version> + <version>3.8.1</version> $ mvn hpi:hpi # mkdir target/classes? # Might I say thanks to github-plugin developers to be the only ones with a usable README.md as their front page! Saved me a lot of mvn grief Grab target/*.hpi file and upload it to your jenkins via Manage Pluggins->Advanced Github-api-plugin Curiously, kohsuke of CloudBees Inc. already is using okhttp 3.9 https://github.com/kohsuke/github-api/blob/master/pom.xml#L157 $ git checkout github-api-1.90 $ vim pom.xml {{ <groupId>org.kohsuke</groupId>}} {{ <artifactId>github-api</artifactId>}} - <version>1.90</version> + <version>1.93</version> ... {{ }} <dependency> - <groupId>com.squareup.okhttp</groupId> + <groupId>com.squareup.okhttp3</groupId> {{ <artifactId>okhttp-urlconnection</artifactId>}} - <version>2.7.5</version> + <version>3.8.1</version> $ mvn hpi:hpi $ mvn install -DskipTests # install locally, needed by github branch-source Grab target/*.hpi file and upload it to your jenkins via Manage Pluggins->Advanced Github-branch-source-plugin $ git checkout github-branch-source-2.3.4 $ vim pom.xml # resolving build rule failure by explicitly specifying newer package at root + <dependency> + <groupId>org.apache.commons</groupId> + <artifactId>commons-lang3</artifactId> + <version>3.7</version> + </dependency> $ vim src/main/java/org/jenkinsci/plugins/github_branch_source/Connector.java -import com.squareup.okhttp.OkHttpClient; -import com.squareup.okhttp.OkUrlFactory; +import okhttp3.OkHttpClient; +import okhttp3.OkUrlFactory; ... -import org.kohsuke.github.extras.OkHttpConnector; +import org.kohsuke.github.extras.OkHttp3Connector; ... - OkHttpClient client = new OkHttpClient().setProxy(getProxy(host)); + OkHttpClient client = new OkHttpClient.Builder().proxy(getProxy(host)).build(); - gb.withConnector(new OkHttpConnector(new OkUrlFactory(client))); + gb.withConnector(new OkHttp3Connector(new OkUrlFactory(client))); $ mvn compile $ mvn hpi:hpi Grab target/*.hpi file and upload it to your jenkins via Manage Pluggins->Advanced Now, I am able to index my Github Enterprise branches! PS: Jenkins was started with ./ibm-java-s390x-80/bin/java -Dcom.ibm.jsse2.overrideDefaultProtocol=TLSv12 -Djavax.net.debug=ssl,handshake,data,verbose, -Djava.library.path=/usr/lib/jni -Dcom.ibm.jsse2.overrideDefaultTLS=true -jar jenkins.war I haven't bothered removing any of the tracing and override options as of yet to see if they have any effect on the fix.

Mark Waite added a comment - 2020-04-20 22:39

Jenkins now also supports JDK 11. AdoptOpenJDK 11 on z390x provides a just in time compiler that should provide good performance with Jenkins without the requirement to upgrade okhttp to a newer version in the various Jenkins components.

Mark Waite added a comment - 2020-04-20 22:39 Jenkins now also supports JDK 11. AdoptOpenJDK 11 on z390x provides a just in time compiler that should provide good performance with Jenkins without the requirement to upgrade okhttp to a newer version in the various Jenkins components.

Liam Newman added a comment - 2020-06-15 21:03

vpaprots
While github-api allowed the use of okhttp3, the github-branch-source plugin has only just this week moved to using okhttp3.
On the plus side, that means this issue is fixed in github-branch-source 2.8.2 .

Liam Newman added a comment - 2020-06-15 21:03 vpaprots While github-api allowed the use of okhttp3, the github-branch-source plugin has only just this week moved to using okhttp3. On the plus side, that means this issue is fixed in github-branch-source 2.8.2 .

Details

Description

Attachments

Activity