Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-73351

io.fabric8.kubernetes.client.KubernetesClientException: Received 401 on websocket . Message: Unauthorized.

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • kubernetes-plugin
    • None

      Jenkins Master: 2.440.3

      kubernetes-plugin: 4246.v5a_12b_1fe120e

      Kubernetes Client API: 6.10.0-240.v57880ce8b_0b_2

      RKE2 version: v1.25.4+rke2r1    

      Jnlp container: `jenkins/inbound-agent:latest-jdk17`

       

      Current setup

      • Jenkins Master deployed in rke2 cluster  in statefulset mode 
      • jnlp agents are running in the same cluster as the controller , same namespace
      • RKE2 based on ubuntu 22.04 
      • We're using nginx ingress controller
      • All infra runs on vmware
      • This is the list of plugins:
      Jenkins: 2.440.3
      OS: Linux - 5.15.0-105-generic
      Java: 17.0.10 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)
      ---
      PrioritySorter:5.0.0
      ace-editor:1.1
      active-directory:2.31
      analysis-model-api:11.6.0
      ansicolor:1.0.3
      ant:497.v94e7d9fffa_b_9
      antisamy-markup-formatter:162.v0e6ec0fcfcf6
      apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
      apache-httpcomponents-client-5-api:5.2.1-1.0
      artifactory:3.18.9
      asm-api:9.6-3.v2e1fa_b_338cd7
      async-http-client:1.9.40.0
      authentication-tokens:1.53.v1c90fd9191a_b_
      authorize-project:1.7.1
      aws-java-sdk:1.12.529-406.vdeff15e5817d
      aws-java-sdk-cloudformation:1.12.529-406.vdeff15e5817d
      aws-java-sdk-codebuild:1.12.529-406.vdeff15e5817d
      aws-java-sdk-ec2:1.12.529-406.vdeff15e5817d
      aws-java-sdk-ecr:1.12.529-406.vdeff15e5817d
      aws-java-sdk-ecs:1.12.529-406.vdeff15e5817d
      aws-java-sdk-efs:1.12.529-406.vdeff15e5817d
      aws-java-sdk-elasticbeanstalk:1.12.529-406.vdeff15e5817d
      aws-java-sdk-iam:1.12.529-406.vdeff15e5817d
      aws-java-sdk-kinesis:1.12.529-406.vdeff15e5817d
      aws-java-sdk-logs:1.12.529-406.vdeff15e5817d
      aws-java-sdk-minimal:1.12.529-406.vdeff15e5817d
      aws-java-sdk-secretsmanager:1.12.529-406.vdeff15e5817d
      aws-java-sdk-sns:1.12.529-406.vdeff15e5817d
      aws-java-sdk-sqs:1.12.529-406.vdeff15e5817d
      aws-java-sdk-ssm:1.12.529-406.vdeff15e5817d
      badge:1.9.1
      basic-branch-build-strategies:81.v05e333931c7d
      bitbucket:223.vd12f2bca5430
      blueocean:1.27.6
      blueocean-autofavorite:1.2.5
      blueocean-bitbucket-pipeline:1.27.6
      blueocean-commons:1.27.6
      blueocean-config:1.27.6
      blueocean-core-js:1.27.6
      blueocean-dashboard:1.27.6
      blueocean-display-url:2.4.2
      blueocean-events:1.27.6
      blueocean-executor-info:1.27.6
      blueocean-git-pipeline:1.27.6
      blueocean-github-pipeline:1.27.6
      blueocean-i18n:1.27.6
      blueocean-jira:1.27.6
      blueocean-jwt:1.27.6
      blueocean-personalization:1.27.6
      blueocean-pipeline-api-impl:1.27.6
      blueocean-pipeline-editor:1.27.6
      blueocean-pipeline-scm-api:1.27.6
      blueocean-rest:1.27.6
      blueocean-rest-impl:1.27.6
      blueocean-web:1.27.6
      bootstrap4-api:4.6.0-6
      bootstrap5-api:5.3.2-2
      bouncycastle-api:2.30.1.77-225.v26ea_c9455fd9
      branch-api:2.1122.v09cb_8ea_8a_724
      build-failure-analyzer:2.4.1
      build-pipeline-plugin:1.5.8
      build-timeout:1.31
      build-user-vars-plugin:1.9
      built-on-column:1.4
      caffeine-api:3.1.8-133.v17b_1ff2e0599
      checks-api:2.0.2
      claim:516.v36293563731d
      cloud-stats:316.vd6d6b_292238d
      cloudbees-bitbucket-branch-source:825.va_6a_dc46a_f97d
      cloudbees-disk-usage-simple:182.v62ca_0c992a_f3
      cloudbees-folder:6.858.v898218f3609d
      cobertura:1.17
      code-coverage-api:4.7.0
      command-launcher:107.v773860566e2e
      commons-lang3-api:3.13.0-62.v7d18e55f51e2
      commons-text-api:1.10.0-78.v3e7b_ea_d5a_fe1
      conditional-buildstep:1.4.3
      config-file-provider:953.v0432a_802e4d2
      configuration-as-code:1670.v564dc8b_982d0
      configurationslicing:548.ve92d48e66b_f8
      copyartifact:714.v28a_34f8c563f
      credentials:1319.v7eb_51b_3a_c97b_
      credentials-binding:657.v2b_19db_7d6e6d
      dashboard-view:2.508.va_74654f026d1
      data-tables-api:1.13.5-1
      dependency-track:4.3.1
      display-url-api:2.200.vb_9327d658781
      docker-commons:439.va_3cb_0a_6a_fb_29
      docker-java-api:3.3.1-79.v20b_53427e041
      docker-plugin:1.4
      docker-workflow:572.v950f58993843
      downstream-build-cache:1.7
      dtkit-api:3.0.2
      durable-task:550.v0930093c4b_a_6
      echarts-api:5.4.0-7
      email-ext:2.100
      envinject:2.908.v66a_774b_31d93
      envinject-api:1.199.v3ce31253ed13
      export-params:1.9
      external-monitor-job:207.v98a_a_37a_85525
      favorite:2.4.3
      file-operations:214.v2e7dc7f25757
      flexible-publish:0.16.1
      font-awesome-api:6.4.2-1
      forensics-api:2.3.0
      fortify:22.2.39
      gerrit-trigger:2.39.1
      git:5.2.0
      git-client:4.4.0
      git-server:99.va_0826a_b_cdfa_d
      github:1.37.3
      github-api:1.314-431.v78d72a_3fe4c3
      github-branch-source:1732.v3f1889a_c475b_
      github-organization-folder:1.6
      gitlab-plugin:1.7.15
      google-oauth-plugin:1.0.9
      gradle:2.8.2
      groovy:453.vcdb_a_c5c99890
      handlebars:3.0.8
      handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953
      hashicorp-vault-plugin:368.v48134f694db_f
      htmlpublisher:1.32
      icon-shim:3.0.0
      instance-identity:173.va_37c494ec4e5
      ionicons-api:56.v1b_1c8c49374e
      ivy:2.5
      jackson2-api:2.16.1-373.ve709c6871598
      jacoco:3.3.6
      jakarta-activation-api:2.0.1-3
      jakarta-mail-api:2.0.1-3
      javadoc:243.vb_b_503b_b_45537
      javax-activation-api:1.2.0-6
      javax-mail-api:1.6.2-9
      jaxb:2.3.9-1
      jdk-tool:73.vddf737284550
      jenkins-design-language:1.27.6
      jenkins-multijob-plugin:623.v03401733c9a_9
      jersey2-api:2.40-1
      jira:3.10
      jjwt-api:0.11.5-77.v646c772fddb_0
      job-dsl:1.87
      job-restrictions:0.8
      jobConfigHistory:1227.v7a_79fc4dc01f
      joda-time-api:2.12.5-5.v5495a_235fedf
      jquery:1.12.4-1
      jquery-ui:1.0.2
      jquery3-api:3.7.1-1
      jsch:0.2.8-65.v052c39de79b_2
      junit:1252.vfc2e5efa_294f
      jython:1.9
      kubernetes:4246.v5a_12b_1fe120e
      kubernetes-client-api:6.10.0-240.v57880ce8b_0b_2
      kubernetes-credentials:0.11
      ldap:694.vc02a_69c9787f
      locale:314.v22ce953dfe9e
      lockable-resources:1185.v0c528656ce04
      logfilesizechecker:1.5
      logstash:2.5.0205.vd05825ed46bd
      mailer:463.vedf8358e006b_
      managed-scripts:1.5.6
      mapdb-api:1.0.9-28.vf251ce40855d
      mask-passwords:150.vf80d33113e80
      matrix-auth:3.1.10
      matrix-project:822.v01b_8c85d16d2
      maven-plugin:3.23
      mercurial:1260.vdfb_723cdcc81
      metrics:4.2.21-449.v6960d7c54c69
      mina-sshd-api-common:2.10.0-69.v28e3e36d18eb_
      mina-sshd-api-core:2.10.0-69.v28e3e36d18eb_
      momentjs:1.1.1
      mq-notifier:1.3.1
      mstest:1.0.1
      multiple-scms:0.8
      muuri-api:0.9.5-3
      naginator:1.19.2
      nant:248.vcc8a_3eec8db_a
      node-iterator-api:49.v58a_8b_35f8363
      nodejs:1.6.1
      oauth-credentials:0.645.ve666a_c332668
      okhttp-api:4.11.0-157.v6852a_a_fa_ec11
      p4:1.14.2
      pam-auth:1.10
      parameterized-trigger:2.46
      pipeline-aggregator-view:1.15
      pipeline-build-step:505.v5f0844d8d126
      pipeline-github-lib:42.v0739460cda_c4
      pipeline-graph-analysis:202.va_d268e64deb_3
      pipeline-groovy-lib:671.v07c339c842e8
      pipeline-input-step:477.v339683a_8d55e
      pipeline-milestone-step:111.v449306f708b_7
      pipeline-model-api:2.2175.v76a_fff0a_2618
      pipeline-model-declarative-agent:1.1.1
      pipeline-model-definition:2.2144.v077a_d1928a_40
      pipeline-model-extensions:2.2175.v76a_fff0a_2618
      pipeline-rest-api:2.33
      pipeline-stage-step:305.ve96d0205c1c6
      pipeline-stage-tags-metadata:2.2144.v077a_d1928a_40
      pipeline-stage-view:2.33
      pipeline-utility-steps:2.16.0
      plain-credentials:143.v1b_df8b_d3b_e48
      plugin-usage-plugin:4.1
      plugin-util-api:3.6.0
      popper-api:1.16.1-3
      popper2-api:2.11.6-2
      powershell:2.0
      prism-api:1.29.0-7
      project-inheritance:21.04.03
      prometheus:2.2.3
      promoted-builds:892.vd6219fc0a_efb
      pubsub-light:1.17
      pull-request-monitoring:1.7.8
      python:1.3
      rabbitmq-consumer:2.8
      rebuild:320.v5a_0933a_e7d61
      resource-disposer:0.23
      robot:3.4.0
      rocketchatnotifier:1.5.2
      role-strategy:680.v3a_6a_1698b_864
      run-condition:1.6
      scm-api:676.v886669a_199a_a_
      script-security:1313.v7a_6067dc7087
      select2-api:4.0.13-8
      simple-theme-plugin:160.vb_76454b_67900
      slack:684.v833089650554
      snakeyaml-api:2.2-111.vc6598e30cc65
      sonar:2.15
      sse-gateway:1.26
      ssh-agent:333.v878b_53c89511
      ssh-credentials:308.ve4497b_ccd8f4
      ssh-slaves:2.916.vd17b_43357ce4
      sshd:3.312.v1c601b_c83b_0e
      structs:337.v1b_04ea_4df7c8
      subversion:2.17.3
      support-core:1371.v709cef748b_d0
      theme-manager:211.vef2a_42c645a_b_
      thinBackup:1.18
      timestamper:1.26
      token-macro:384.vf35b_f26814ec
      trilead-api:2.84.v72119de229b_7
      variant:60.v7290fc0eb_b_cd
      vsphere-cloud:2.27
      warnings-ng:10.4.0
      windows-slaves:1.8.1
      workflow-aggregator:596.v8c21c963d92d
      workflow-api:1291.v51fd2a_625da_7
      workflow-basic-steps:1042.ve7b_140c4a_e0c
      workflow-cps:3880.vb_ef4b_5cfd270
      workflow-cps-global-lib:609.vd95673f149b_b
      workflow-durable-task-step:1331.vc8c2fed35334
      workflow-job:1385.vb_58b_86ea_fff1
      workflow-multibranch:756.v891d88f2cd46
      workflow-scm-step:415.v434365564324
      workflow-step-api:657.v03b_e8115821b_
      workflow-support:865.v43e78cc44e0d
      ws-cleanup:0.45
      xunit:3.1.3 

       

      Problem Statement

      Jenkins agent pod containing 9 to 11 containers occasionally run into the following error :

      io.fabric8.kubernetes.client.KubernetesClientException: Received 401 on websocket. Failure executing: GET at: https://kubernetes.default/api/v1/namespaces/jenkins/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dxxxxxxxxxxxxxxxxxx-feature-2fdevops-develop-demo-8-dbt8g--9p398&resourceVersion=70196475&timeoutSeconds=600&watch=true. Message: Unauthorized. 

      =>the issue occured after an upgrade from 2.375.3 LTS to 2.440.3 LTS

          [JENKINS-73351] io.fabric8.kubernetes.client.KubernetesClientException: Received 401 on websocket . Message: Unauthorized.

          Mark Waite added a comment -

          This looks more like a local scalability issue than a Jenkins bug. The Jenkins project runs hundreds of kubernetes agents concurrently when testing the Jenkins plugin bill of materials.

          There is a better chance of suggestions to scalability issues on community.jenkins.io or on the Jenkins user mailing list. More people read those locations than read the Jenkins issue tracker.

          Mark Waite added a comment - This looks more like a local scalability issue than a Jenkins bug. The Jenkins project runs hundreds of kubernetes agents concurrently when testing the Jenkins plugin bill of materials. There is a better chance of suggestions to scalability issues on community.jenkins.io or on the Jenkins user mailing list. More people read those locations than read the Jenkins issue tracker.

          engel80 added a comment -

          I'm getting the same error after the Jenkins upgrade, I have a different controller version and it seems to be a bug in the Kubernetes plugin.

          Jenkins Controller: 2.452.2

          kubernetes-plugin: 4246.v5a_12b_1fe120e

          Kubernetes Client API: 6.10.0-240.v57880ce8b_0b_2

          engel80 added a comment - I'm getting the same error after the Jenkins upgrade, I have a different controller version and it seems to be a bug in the Kubernetes plugin. Jenkins Controller: 2.452.2 kubernetes-plugin:  4246.v5a_12b_1fe120e Kubernetes Client API:  6.10.0-240.v57880ce8b_0b_2

          Mark Waite added a comment -

          Please provide the configuration details that are requested in "How to report an issue" in hopes of persuading others to investigate the issue. When those details are not provided, anyone that investigates will make their own guesses about the configuration details. Those guesses may differ from your configuration in ways that are important.

          Additional details that may increase the chances of someone investigating it include:

          • What Kubernetes version is running in the environment?
          • What virtual machine types are used for the Kubernetes cluster?
          • What is the configured ingress for the Kubernetes cluster?
          • Are there other details that a Kubernetes expert might need to know when trying to recreate your environment?

          Mark Waite added a comment - Please provide the configuration details that are requested in "How to report an issue" in hopes of persuading others to investigate the issue. When those details are not provided, anyone that investigates will make their own guesses about the configuration details. Those guesses may differ from your configuration in ways that are important. Additional details that may increase the chances of someone investigating it include: What Kubernetes version is running in the environment? What virtual machine types are used for the Kubernetes cluster? What is the configured ingress for the Kubernetes cluster? Are there other details that a Kubernetes expert might need to know when trying to recreate your environment?

          Hello markewaite ,

          more details were provided in the description

           

          BR.

          Mohamed ilyess added a comment - Hello markewaite , more details were provided in the description   BR.

          Edwin added a comment -

          We're seeing the same error. We're running Jenkins on EKS installed by Helm chart and it's connecting to other EKS clusters in other AWS accounts.
          Connecting to those other clusters works fine, but the `watcher` seems to have issues.
          Could it be that the token that is renewed according to this comment:  https://github.com/jenkinsci/kubernetes-plugin/blob/master/src/main/java/org/csanchez/jenkins/plugins/kubernetes/KubernetesClientProvider.java#L33 is not used by the `watchCloud` watcher?

          Helm chart: 5.7.25
          Jenkins: 2.479.2
          Kubernetes plugin: 4306.vc91e951ea_eb_d
          Kubernetes Credentials Plugin: 190.v03c305394deb_
          AWS Credentials Plugin: 240.v6d844a_6f5480
          EKS Token Plugin: 0.0.2

          Edwin added a comment - We're seeing the same error. We're running Jenkins on EKS installed by Helm chart and it's connecting to other EKS clusters in other AWS accounts. Connecting to those other clusters works fine, but the `watcher` seems to have issues. Could it be that the token that is renewed according to this comment:  https://github.com/jenkinsci/kubernetes-plugin/blob/master/src/main/java/org/csanchez/jenkins/plugins/kubernetes/KubernetesClientProvider.java#L33 is not used by the `watchCloud` watcher? Helm chart: 5.7.25 Jenkins: 2.479.2 Kubernetes plugin: 4306.vc91e951ea_eb_d Kubernetes Credentials Plugin: 190.v03c305394deb_ AWS Credentials Plugin: 240.v6d844a_6f5480 EKS Token Plugin: 0.0.2

            Unassigned Unassigned
            elyess_mez Mohamed ilyess
            Votes:
            2 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: