[JENKINS-40294] Exec Failure: HTTP:404. Message:Not Found

Type: Bug
Resolution: Fixed
Priority: Critical
Component/s: kubernetes-plugin
Labels:
- kubernetes
Environment:
Jenkins 2.35, Kubernetes Plugin 0.9, Minikube 0.13.1

Similar Issues:
Powered by SuggestiMate

Show

I'm trying to run a simple pipeline based on the example on the README:

podTemplate(label: 'mypod', containers: [
    containerTemplate(name: 'maven', image: 'maven:3.3.9-jdk-8-alpine', ttyEnabled: true, command: 'cat'),
    containerTemplate(name: 'jnlp', image: 'jenkinsci/jnlp-slave', args: '${computer.jnlpmac} ${computer.name}'),
]) {

    node ('mypod') {
        stage 'Get a Maven project'
        git 'https://github.com/jenkinsci/kubernetes-plugin.git'
        container('maven') {
            stage 'Build a Maven project'
            sh 'mvn clean install'
        }
    }
}

The pod is created and started, but when Jenkins tries to run the shell script for the build, I get:

[kubernetes-plugin-pipeline-example] Running shell script
Executing shell script inside container [maven] of pod [kubernetes-f914ee7f49c44f7eaa182af5fda74c93-60beb1f2a295]
java.net.ProtocolException: Expected HTTP 101 response but was '404 Not Found'
	at com.squareup.okhttp.ws.WebSocketCall.createWebSocket(WebSocketCall.java:123)
	at com.squareup.okhttp.ws.WebSocketCall.access$000(WebSocketCall.java:40)
	at com.squareup.okhttp.ws.WebSocketCall$1.onResponse(WebSocketCall.java:98)
	at com.squareup.okhttp.Call$AsyncCall.execute(Call.java:177)
	at com.squareup.okhttp.internal.NamedRunnable.run(NamedRunnable.java:33)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

The relevant section from Jenkins' log is attached. I increased the log verbosity of io.fabric8.kubernetes, org.csanchez.jenkins.plugins.kubernetes, com.squareup.okhttp and org.apache.http to ALL but it doesn't show which HTTP request got a 404 response.

- - Sort By Name
  - Sort By Date
  - Ascending
  - Descending
  - Thumbnails
  - List
  - Download All

image-2019-09-18-14-04-48-431.png
225 kB
2019-09-18 06:04
jenkins-kubernetes.log
18 kB
2016-12-07 20:47

Daniel Serodio created issue - 2016-12-07 20:51

Jean Neveux added a comment - 2016-12-10 21:08

I'm having the same exception with Kubernet version:

Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.4", GitCommit:"3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56", GitTreeState:"clean", BuildDate:"2016-10-21T02:48:38Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.4", GitCommit:"3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56", GitTreeState:"clean", BuildDate:"2016-10-21T02:42:39Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

And I'm also having another exception at the end of the script:

io.fabric8.kubernetes.client.KubernetesClientException: Not Found
	at io.fabric8.kubernetes.client.dsl.internal.ExecWebSocketListener.onFailure(ExecWebSocketListener.java:159)
	at com.squareup.okhttp.ws.WebSocketCall$1.onResponse(WebSocketCall.java:100)
	at com.squareup.okhttp.Call$AsyncCall.execute(Call.java:177)
	at com.squareup.okhttp.internal.NamedRunnable.run(NamedRunnable.java:33)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Finished: FAILURE

Jean Neveux added a comment - 2016-12-10 21:08 I'm having the same exception with Kubernet version: Client Version: version.Info{Major: "1" , Minor: "4" , GitVersion: "v1.4.4" , GitCommit: "3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56" , GitTreeState: "clean" , BuildDate: "2016-10-21T02:48:38Z" , GoVersion: "go1.6.3" , Compiler : "gc" , Platform: "linux/amd64" } Server Version: version.Info{Major: "1" , Minor: "4" , GitVersion: "v1.4.4" , GitCommit: "3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56" , GitTreeState: "clean" , BuildDate: "2016-10-21T02:42:39Z" , GoVersion: "go1.6.3" , Compiler : "gc" , Platform: "linux/amd64" } And I'm also having another exception at the end of the script: io.fabric8.kubernetes.client.KubernetesClientException: Not Found at io.fabric8.kubernetes.client.dsl.internal.ExecWebSocketListener.onFailure(ExecWebSocketListener.java:159) at com.squareup.okhttp.ws.WebSocketCall$1.onResponse(WebSocketCall.java:100) at com.squareup.okhttp.Call$AsyncCall.execute(Call.java:177) at com.squareup.okhttp.internal.NamedRunnable.run(NamedRunnable.java:33) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:745) Finished: FAILURE

Val Orekhov added a comment - 2016-12-13 01:56

I'm having an exact same issue on a K8N cluster configured on bare metal. Tried configuring a custom logger on org.apache.http to capture the calls but it's not showing any entries. Were you able to figure out a solution to this problem?
Thanks!
Val

Val Orekhov added a comment - 2016-12-13 01:56 I'm having an exact same issue on a K8N cluster configured on bare metal. Tried configuring a custom logger on org.apache.http to capture the calls but it's not showing any entries. Were you able to figure out a solution to this problem? Thanks! Val

Val Orekhov made changes - 2016-12-13 02:10

Priority

Original: Minor [ 4 ]

New: Critical [ 2 ]

Carlos Sanchez added a comment - 2016-12-18 19:27

are there any errors in the slave log? under /computer/xxx/log

Carlos Sanchez added a comment - 2016-12-18 19:27 are there any errors in the slave log? under /computer/xxx/log

Jesse Redl added a comment - 2016-12-21 23:56

csanchez I also am encountering this exact error. I pulled the following logs out of my GKE logs:

Dec 21, 2016 11:51:37 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Excess workload after pending Spot instances: 1
Dec 21, 2016 11:51:37 PM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
INFO: Started provisioning Kubernetes Pod Template from kubernetes with 1 executors. Remaining excess workload: 0
Dec 21, 2016 11:51:37 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback call
INFO: Created Pod: default-11ea535e917446
Dec 21, 2016 11:51:37 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback call
INFO: Waiting for Pod to be scheduled (0/100): default-11ea535e917446
Dec 21, 2016 11:51:39 PM hudson.TcpSlaveAgentListener$ConnectionHandler run
INFO: Accepted connection #21 from /10.156.3.229:38600
Dec 21, 2016 11:51:39 PM hudson.TcpSlaveAgentListener$ConnectionHandler error
WARNING: Connection #21 is aborted: Disabled protocol:Protocol:JNLP3-connect
Dec 21, 2016 11:51:39 PM hudson.TcpSlaveAgentListener$ConnectionHandler run
INFO: Accepted connection #22 from /10.156.3.229:38602
Dec 21, 2016 11:51:43 PM io.fabric8.kubernetes.client.dsl.internal.ExecWebSocketListener onFailure
SEVERE: Exec Failure: HTTP:400. Message:Bad Request
java.net.ProtocolException: Expected HTTP 101 response but was '400 Bad Request'
	at com.squareup.okhttp.ws.WebSocketCall.createWebSocket(WebSocketCall.java:123)
	at com.squareup.okhttp.ws.WebSocketCall.access$000(WebSocketCall.java:40)
	at com.squareup.okhttp.ws.WebSocketCall$1.onResponse(WebSocketCall.java:98)
	at com.squareup.okhttp.Call$AsyncCall.execute(Call.java:177)
	at com.squareup.okhttp.internal.NamedRunnable.run(NamedRunnable.java:33)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Dec 21, 2016 11:51:43 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
INFO: Terminating Kubernetes instance for slave default-11ea535e917446
Dec 21, 2016 11:51:43 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
INFO: Terminated Kubernetes instance for slave default-11ea535e917446
Dec 21, 2016 11:51:43 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
INFO: Disconnected computer default-11ea535e917446
Dec 21, 2016 11:51:43 PM org.jenkinsci.plugins.workflow.job.WorkflowRun finish
INFO: vendasta/AA/jenkins-build #58 completed: FAILURE
Dec 21, 2016 11:51:43 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback call
SEVERE: Error in provisioning; slave=KubernetesSlave name: default-11ea535e917446, template=org.csanchez.jenkins.plugins.kubernetes.PodTemplate@28a6c5c2
java.lang.IllegalStateException: Node was deleted, computer is null
	at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback.call(KubernetesCloud.java:556)
	at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback.call(KubernetesCloud.java:482)
	at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Dec 21, 2016 11:51:47 PM hudson.slaves.NodeProvisioner$2 run
WARNING: Provisioned agent Kubernetes Pod Template failed to launch
java.lang.IllegalStateException: Node was deleted, computer is null
	at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback.call(KubernetesCloud.java:556)
	at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback.call(KubernetesCloud.java:482)
	at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Jesse Redl added a comment - 2016-12-21 23:56 csanchez I also am encountering this exact error. I pulled the following logs out of my GKE logs: Dec 21, 2016 11:51:37 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision INFO: Excess workload after pending Spot instances: 1 Dec 21, 2016 11:51:37 PM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply INFO: Started provisioning Kubernetes Pod Template from kubernetes with 1 executors. Remaining excess workload: 0 Dec 21, 2016 11:51:37 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback call INFO: Created Pod: default -11ea535e917446 Dec 21, 2016 11:51:37 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback call INFO: Waiting for Pod to be scheduled (0/100): default -11ea535e917446 Dec 21, 2016 11:51:39 PM hudson.TcpSlaveAgentListener$ConnectionHandler run INFO: Accepted connection #21 from /10.156.3.229:38600 Dec 21, 2016 11:51:39 PM hudson.TcpSlaveAgentListener$ConnectionHandler error WARNING: Connection #21 is aborted: Disabled protocol:Protocol:JNLP3-connect Dec 21, 2016 11:51:39 PM hudson.TcpSlaveAgentListener$ConnectionHandler run INFO: Accepted connection #22 from /10.156.3.229:38602 Dec 21, 2016 11:51:43 PM io.fabric8.kubernetes.client.dsl.internal.ExecWebSocketListener onFailure SEVERE: Exec Failure: HTTP:400. Message:Bad Request java.net.ProtocolException: Expected HTTP 101 response but was '400 Bad Request' at com.squareup.okhttp.ws.WebSocketCall.createWebSocket(WebSocketCall.java:123) at com.squareup.okhttp.ws.WebSocketCall.access$000(WebSocketCall.java:40) at com.squareup.okhttp.ws.WebSocketCall$1.onResponse(WebSocketCall.java:98) at com.squareup.okhttp.Call$AsyncCall.execute(Call.java:177) at com.squareup.okhttp.internal.NamedRunnable.run(NamedRunnable.java:33) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:745) Dec 21, 2016 11:51:43 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate INFO: Terminating Kubernetes instance for slave default -11ea535e917446 Dec 21, 2016 11:51:43 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate INFO: Terminated Kubernetes instance for slave default -11ea535e917446 Dec 21, 2016 11:51:43 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate INFO: Disconnected computer default -11ea535e917446 Dec 21, 2016 11:51:43 PM org.jenkinsci.plugins.workflow.job.WorkflowRun finish INFO: vendasta/AA/jenkins-build #58 completed: FAILURE Dec 21, 2016 11:51:43 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback call SEVERE: Error in provisioning; slave=KubernetesSlave name: default -11ea535e917446, template=org.csanchez.jenkins.plugins.kubernetes.PodTemplate@28a6c5c2 java.lang.IllegalStateException: Node was deleted, computer is null at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback.call(KubernetesCloud.java:556) at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback.call(KubernetesCloud.java:482) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:745) Dec 21, 2016 11:51:47 PM hudson.slaves.NodeProvisioner$2 run WARNING: Provisioned agent Kubernetes Pod Template failed to launch java.lang.IllegalStateException: Node was deleted, computer is null at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback.call(KubernetesCloud.java:556) at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback.call(KubernetesCloud.java:482) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:745)

Carlos Sanchez added a comment - 2016-12-22 07:55

as I said, is there anything in the slave logs in jenkins web ui under /computer/xxx/log ?

Carlos Sanchez added a comment - 2016-12-22 07:55 as I said, is there anything in the slave logs in jenkins web ui under /computer/xxx/log ?

Todd Lyons added a comment - 2017-10-04 21:23

We got bit with this today when creating a new Jenkinsfile. The issue appears to be when you use:

node('mypod')

If we just change it to "mypods" or use a dynamic label such as:

def pod_label = "buildpod-${env.JOB_NAME}-${env.BUILD_NUMBER}".replace('_', '-').replace('/', '-')

(and of course use the pod_label as the node selector), then it seems to work fine.

This is what the labels are on this node when the jnlp slave is running:

jenkins-slave-jkznb-rgzq9          2/2       Terminating   0          42s       jenkins/mypod=true,jenkins=slave

It is choking on the node selector "mypod". Unknown why.

Todd Lyons added a comment - 2017-10-04 21:23 We got bit with this today when creating a new Jenkinsfile. The issue appears to be when you use: node( 'mypod' ) If we just change it to "mypods" or use a dynamic label such as: def pod_label = "buildpod-${env.JOB_NAME}-${env.BUILD_NUMBER}" .replace( '_' , '-' ).replace( '/' , '-' ) (and of course use the pod_label as the node selector), then it seems to work fine. This is what the labels are on this node when the jnlp slave is running: jenkins-slave-jkznb-rgzq9 2/2 Terminating 0 42s jenkins/mypod= true ,jenkins=slave It is choking on the node selector "mypod". Unknown why.

Carlos Sanchez added a comment - 2018-05-25 09:21

so your pipeline is probably getting a different pod template because you have one in the global config with the same label. And that template doesn't have the container you are trying to exec into

Carlos Sanchez added a comment - 2018-05-25 09:21 so your pipeline is probably getting a different pod template because you have one in the global config with the same label. And that template doesn't have the container you are trying to exec into

Carlos Sanchez made changes - 2018-05-25 09:21

Resolution		New: Not A Defect [ 7 ]
Status	Original: Open [ 1 ]	New: Resolved [ 5 ]

Assignee:: Carlos Sanchez

Reporter:: Daniel Serodio

Votes:: 1 Vote for this issue

Watchers:: 9 Start watching this issue

Created:: 2016-12-07 20:51

Updated:: 2020-11-17 10:07

Resolved:: 2020-11-17 10:07

Jenkins

Details

Description

Attachments

Attachments

Activity

Collapse comment: Jean Neveux added a comment - 2016-12-10 21:08

Expand comment: Jean Neveux added a comment - 2016-12-10 21:08

Collapse comment: Val Orekhov added a comment - 2016-12-13 01:56

Expand comment: Val Orekhov added a comment - 2016-12-13 01:56

Collapse comment: Carlos Sanchez added a comment - 2016-12-18 19:27

Expand comment: Carlos Sanchez added a comment - 2016-12-18 19:27

Collapse comment: Jesse Redl added a comment - 2016-12-21 23:56

Expand comment: Jesse Redl added a comment - 2016-12-21 23:56

Collapse comment: Carlos Sanchez added a comment - 2016-12-22 07:55

Expand comment: Carlos Sanchez added a comment - 2016-12-22 07:55

Collapse comment: Todd Lyons added a comment - 2017-10-04 21:23

Expand comment: Todd Lyons added a comment - 2017-10-04 21:23

Collapse comment: Carlos Sanchez added a comment - 2018-05-25 09:21

Expand comment: Carlos Sanchez added a comment - 2018-05-25 09:21

People

Dates