Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-65069

Jenkins 2.263.4 & 2.277.1, GHE api token not working due to SSL issue in Docker image

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Critical Critical
    • docker
    • Debian 10 Buster
      Issue with Jenkins 2.263.4 and 2.277.1

      Updated our jenkins server to 2.263.4 and later today to 2.277.1 and updated all the plugins. Now if i run build with parameters, getting below error. When i check the GHE credentials and test connection, it doesn't work. If i downgrade the version to 2.263.3 everything works normal.

      Major change in 2.263.4 is Debain 10 from Debian 9 and java change.

      sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
      	at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
      	at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
      	at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
      	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:451)
      Caused: sun.security.validator.ValidatorException: PKIX path building failed
      	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:456)
      	at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:323)
      	at sun.security.validator.Validator.validate(Validator.java:271)
      	at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:315)
      	at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:223)
      	at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:129)
      	at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:638)
      Caused: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
      	at sun.security.ssl.Alert.createSSLException(Alert.java:131)
      	at sun.security.ssl.TransportContext.fatal(TransportContext.java:324)
      	at sun.security.ssl.TransportContext.fatal(TransportContext.java:267)
      	at sun.security.ssl.TransportContext.fatal(TransportContext.java:262)
      	at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:654)
      	at sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:473)
      	at sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:369)
      	at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:377)
      	at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444)
      	at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:422)
      	at sun.security.ssl.TransportContext.dispatch(TransportContext.java:182)
      	at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149)
      	at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143)
      	at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054)
      	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394)
      	at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:336)
      	at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:300)
      	at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:185)
      	at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.java:224)
      	at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.java:108)
      	at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.java:88)
      	at okhttp3.internal.connection.Transmitter.newExchange(Transmitter.java:169)
      	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:41)
      	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
      	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
      	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94)
      	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
      	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
      	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
      	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
      	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88)
      	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
      	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
      	at org.kohsuke.github.extras.okhttp3.ObsoleteUrlFactory$UnexpectedException.lambda$static$0(ObsoleteUrlFactory.java:1363)
      	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
      	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
      	at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:229)
      	at okhttp3.RealCall.execute(RealCall.java:81)
      	at org.kohsuke.github.extras.okhttp3.ObsoleteUrlFactory$OkHttpURLConnection.getResponse(ObsoleteUrlFactory.java:669)
      	at org.kohsuke.github.extras.okhttp3.ObsoleteUrlFactory$OkHttpURLConnection.getResponseCode(ObsoleteUrlFactory.java:700)
      	at org.kohsuke.github.extras.okhttp3.ObsoleteUrlFactory$DelegatingHttpsURLConnection.getResponseCode(ObsoleteUrlFactory.java:1062)
      	at org.kohsuke.github.GitHubHttpUrlConnectionClient.getResponseInfo(GitHubHttpUrlConnectionClient.java:64)
      	at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:394)
      Caused: org.kohsuke.github.HttpException: Server returned HTTP response code: -1, message: 'null' for URL: https://repo.west.com/api/v3/rate_limit
      	at org.kohsuke.github.GitHubClient.interpretApiError(GitHubClient.java:494)
      	at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:414)
      	at org.kohsuke.github.GitHubClient.getRateLimit(GitHubClient.java:232)
      	at org.kohsuke.github.GitHubClient.rateLimit(GitHubClient.java:283)
      	at org.kohsuke.github.GitHubRateLimitChecker.checkRateLimit(GitHubRateLimitChecker.java:122)
      	at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:392)
      	at org.kohsuke.github.GitHubClient.fetch(GitHubClient.java:129)
      	at org.kohsuke.github.GitHubClient.checkApiUrlValidity(GitHubClient.java:325)
      	at org.kohsuke.github.GitHub.checkApiUrlValidity(GitHub.java:1195)
      	at org.jenkinsci.plugins.github_branch_source.Connector$GitHubConnection.verifyConnection(Connector.java:678)
      Caused: java.io.IOException: It seems https://repo.west.com/api/v3 is unreachable
      	at org.jenkinsci.plugins.github_branch_source.Connector$GitHubConnection.verifyConnection(Connector.java:681)
      	at org.jenkinsci.plugins.github_branch_source.Connector$GitHubConnection.connect(Connector.java:635)
      	at org.jenkinsci.plugins.github_branch_source.Connector$GitHubConnection.access$200(Connector.java:589)
      	at org.jenkinsci.plugins.github_branch_source.Connector.connect(Connector.java:361)
      	at org.jenkinsci.plugins.github_branch_source.GitHubSCMSource.retrieve(GitHubSCMSource.java:1582)
      	at jenkins.scm.api.SCMSource.fetch(SCMSource.java:582)
      	at org.jenkinsci.plugins.workflow.multibranch.SCMBinder.create(SCMBinder.java:98)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:309)
      	at hudson.model.ResourceController.execute(ResourceController.java:97)
      	at hudson.model.Executor.run(Executor.java:429)
      Finished: FAILURE
      

          [JENKINS-65069] Jenkins 2.263.4 & 2.277.1, GHE api token not working due to SSL issue in Docker image

          Oleg Nenashev added a comment -

          No idea. There was no related changes in the 2.263.4 LTS. could you please confirm that there are no plugin differences after the rollback?

          Oleg Nenashev added a comment - No idea. There was no related changes in the 2.263.4 LTS. could you please confirm that there are no plugin differences after the rollback?

          Hashim added a comment - - edited

          yes I can confirm this is not the issue with any plugin update and after rollback I didn't change the plugin version,plugins are latest version. Issue is with the certificates are not updating i guess, might be the issue with "AdoptOpenJDK".

          Now I updated image version to 2.277.1

          Referred below issues: may be related to this

          https://github.com/AdoptOpenJDK/openjdk-installer/issues/105

          https://github.com/AdoptOpenJDK/openjdk-installer/issues/25

          https://github.com/jitsi/jitsi-meet/issues/8243

           

          Below is my Dockerfile:

          FROM jenkins/jenkins:2.277.1-lts
          
          ENV GIT_MKVER_VERSION=1.2.0
          ENV GIT_CHGLOG_VERSION=0.10.0
          
          # Install packages
          USER root
          RUN apt-get update && apt-get install python-pip createrepo jq file zip -y && apt-get clean \
              && pip install awscli yamllint cerberus ruamel.yaml \
              && curl -SL https://github.com/idc101/git-mkver/releases/download/v${GIT_MKVER_VERSION}/git-mkver-linux-amd64-${GIT_MKVER_VERSION}.tar.gz > git-mkver.tar.gz \
              && tar xzvf git-mkver.tar.gz -C /usr/bin/ \
              && chmod +x /usr/bin/git-mkver \
              && rm -rf git-mkver.tar.gz \
              && curl -SL https://github.com/git-chglog/git-chglog/releases/download/${GIT_CHGLOG_VERSION}/git-chglog_linux_amd64 > /usr/bin/git-chglog \
              && chmod +x /usr/bin/git-chglog
          
          # Install  CA certificates
          COPY *.crt /usr/local/share/ca-certificates/
          RUN chmod 644 /usr/local/share/ca-certificates/corp_*.crt && \
              update-ca-certificates
          USER jenkins
          
          
          # Copy init script
          COPY init.groovy /usr/share/jenkins/ref/init.groovy.d/
          
          # Disable the Installation Wizzard and DirectoryBrowserSupport.CSP
          ENV JAVA_OPTS="-Djenkins.install.runSetupWizard=false -Dio.jenkins.plugins.casc.ConfigurationAsCode.initialDelay=5000 -Dhudson.model.DirectoryBrowserSupport.CSP="
          
          

          Hashim added a comment - - edited yes I can confirm this is not the issue with any plugin update and after rollback I didn't change the plugin version,plugins are latest version. Issue is with the certificates are not updating i guess, might be the issue with "AdoptOpenJDK". Now I updated image version to 2.277.1 Referred below issues: may be related to this https://github.com/AdoptOpenJDK/openjdk-installer/issues/105 https://github.com/AdoptOpenJDK/openjdk-installer/issues/25 https://github.com/jitsi/jitsi-meet/issues/8243   Below is my Dockerfile: FROM jenkins/jenkins:2.277.1-lts ENV GIT_MKVER_VERSION=1.2.0 ENV GIT_CHGLOG_VERSION=0.10.0 # Install packages USER root RUN apt-get update && apt-get install python-pip createrepo jq file zip -y && apt-get clean \ && pip install awscli yamllint cerberus ruamel.yaml \ && curl -SL https: //github.com/idc101/git-mkver/releases/download/v${GIT_MKVER_VERSION}/git-mkver-linux-amd64-${GIT_MKVER_VERSION}.tar.gz > git-mkver.tar.gz \ && tar xzvf git-mkver.tar.gz -C /usr/bin/ \ && chmod +x /usr/bin/git-mkver \ && rm -rf git-mkver.tar.gz \ && curl -SL https: //github.com/git-chglog/git-chglog/releases/download/${GIT_CHGLOG_VERSION}/git-chglog_linux_amd64 > /usr/bin/git-chglog \ && chmod +x /usr/bin/git-chglog # Install CA certificates COPY *.crt /usr/local/share/ca-certificates/ RUN chmod 644 /usr/local/share/ca-certificates/corp_*.crt && \ update-ca-certificates USER jenkins # Copy init script COPY init.groovy /usr/share/jenkins/ref/init.groovy.d/ # Disable the Installation Wizzard and DirectoryBrowserSupport.CSP ENV JAVA_OPTS= "-Djenkins.install.runSetupWizard= false -Dio.jenkins.plugins.casc.ConfigurationAsCode.initialDelay=5000 -Dhudson.model.DirectoryBrowserSupport.CSP="

          Oleg Nenashev added a comment -

          hashim I suggest that you follow the workaround guidelines in https://github.com/jitsi/jitsi-meet/issues/8243#issuecomment-744181944 . It does not look like the fix has got into Eclipse Adoptium yet

          CC markewaite, we might need to update https://www.jenkins.io/doc/upgrade-guide/2.263/#upgrading-to-jenkins-lts-2-263-4 

          Oleg Nenashev added a comment - hashim  I suggest that you follow the workaround guidelines in https://github.com/jitsi/jitsi-meet/issues/8243#issuecomment-744181944  . It does not look like the fix has got into Eclipse Adoptium yet CC markewaite , we might need to update https://www.jenkins.io/doc/upgrade-guide/2.263/#upgrading-to-jenkins-lts-2-263-4  

          Mark Waite added a comment -

          I submitted PR 4168 to add this to the Jenkins 2.263.4 upgrade guide. Phrasing improvements and other refinements are welcomed.

          Mark Waite added a comment - I submitted PR 4168 to add this to the Jenkins 2.263.4 upgrade guide. Phrasing improvements and other refinements are welcomed.

          Hashim added a comment -

          Thank you for the update markewaite & oleg_nenashev.

          I did with the command below and that fix the issue for time being.

          /opt/java/openjdk/bin/keytool -import -v -trustcacerts -noprompt -alias auth.HOSTNAME -file /usr/local/share/ca-certificates/xxxxxx.crt -keystore /opt/java/openjdk/jre/lib/security/cacerts -keypass changeit -storepass changeit
          

          Hashim added a comment - Thank you for the update markewaite & oleg_nenashev . I did with the command below and that fix the issue for time being. /opt/java/openjdk/bin/keytool - import -v -trustcacerts -noprompt -alias auth.HOSTNAME -file /usr/local/share/ca-certificates/xxxxxx.crt -keystore /opt/java/openjdk/jre/lib/security/cacerts -keypass changeit -storepass changeit

          Oleg Nenashev added a comment -

          The interesting question is what we do next. Keep it as is in upgrade guidelines? We could also apply a documented fix in init scripts, which automatically imports certificates. It is doable, but I am not sure how we distinguish the legitimate certificates

          Oleg Nenashev added a comment - The interesting question is what we do next. Keep it as is in upgrade guidelines? We could also apply a documented fix in init scripts, which automatically imports certificates. It is doable, but I am not sure how we distinguish the legitimate certificates

            Unassigned Unassigned
            hashim Hashim
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: