-
Bug
-
Resolution: Duplicate
-
Blocker
-
None
-
HideWindows 7, Jenkins 2.95 -> 2.99
I have about 5-7 Slaves connecting to the Jenkins server (using JNLP 1 - since I am in a safe business intranet area)
CentOS root, macOS Agents connecting to the root node with JNLP4.
Workaround: Update Remoting on the agent side to 3.15ShowWindows 7, Jenkins 2.95 -> 2.99 I have about 5-7 Slaves connecting to the Jenkins server (using JNLP 1 - since I am in a safe business intranet area) CentOS root, macOS Agents connecting to the root node with JNLP4. Workaround: Update Remoting on the agent side to 3.15
-
Powered by SuggestiMate
After having installed Jenkins Version 2.99, none of my slaves did connect to the server anymore. Even a restart of the java script on the slave did not help.
Workaround: Update Remoting on the agent side to 3.15
- is related to
-
JENKINS-48766 Jenkins Core should provide info about minimum supported Remoting version in API/REST API/logs
-
- Resolved
-
-
JENKINS-48761 Jenkins Agents do not connect after update from version 2.95 to version 2.99 and jobs fail
-
- Closed
-
- links to
[JENKINS-48754] Jenkins Slaves do not connect after update from version 2.95 to version 2.99
For me : I've uninstalled my Windows Service agent and click again on "Launch agent from browser". I guess it's the latest with this method ?
In the agent.jar manifest file it's written 3.15 for the version...
My first guess is that Remoting receiver thread is not active due to whatever reason. Or maybe not, the HTTPs receiver should be working
Please provide...
1) Jenkins log, starting from the master startup
2) Thread dump for the master
If you could just generate a bundle using https://wiki.jenkins.io/display/JENKINS/Support+Core+Plugin, it would be great
Some investigation while I am waiting for the info...
1) No changes in JnlpAgentEndpoint since 2.95
2) No changes in Engine#connect()
3) AFAICT https://github.com/jenkinsci/remoting/pull/210 is the most plausible source of regressions, but I do not see the bogus logic so far.
Please also provide agent startup arguments
A full support bundle would help though. In particular I am interested whether the agent uses HTTPS and what are the certificate check flags
Yes, for JNLP agent I really need startup arguments.
Other info:
- 8081 is a TCP agent listener port
- TCP Agent Listener thread is running
- There is a proxy configured on the instance. Likely it's not related since there are other community votes
"TCP agent listener port=8081" id=53 (0x35) state=RUNNABLE cpu=0% (running in native) at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) at hudson.TcpSlaveAgentListener.run(TcpSlaveAgentListener.java:157)
OK, How can I retrieve agent startup arguments ?
On startup page for the agent start through the browser I have
java -jar agent.jar -jnlpUrl http://myserver:10004/computer/Windows%20Slave/slave-agent.jnlp -secret **************************************************************** -workDir "D:\Jenkins"
(Note: I've replaced the real server name by myserver, and changed the secret code).
Seeing something very similar.
In a job:
First time build. Skipping changelog. No emails were triggered. FATAL: command execution failed Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from mdbuildserver01217.gamesys.corp/10.194.194.147:60268 at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1696) at hudson.remoting.UserResponse.retrieve(UserRequest.java:313) at hudson.remoting.Channel.call(Channel.java:909) at hudson.Launcher$RemoteLauncher.launch(Launcher.java:1053) at hudson.Launcher$ProcStarter.start(Launcher.java:450) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:109) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744) at hudson.model.Build$BuildExecution.build(Build.java:206) at hudson.model.Build$BuildExecution.doRun(Build.java:163) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504) at hudson.model.Run.execute(Run.java:1727) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429) java.lang.NoSuchMethodError: hudson.Launcher$RemoteLaunchCallable.getOpenChannelOrFail()Lhudson/remoting/Channel; at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1292) at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1260) at hudson.remoting.UserRequest.perform(UserRequest.java:207) at hudson.remoting.UserRequest.perform(UserRequest.java:53) at hudson.remoting.Request$2.run(Request.java:358) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at hudson.remoting.Engine$1$1.run(Engine.java:98) at java.lang.Thread.run(Thread.java:748) Caused: java.io.IOException: Remote call on JNLP4-connect connection from mdbuildserver01217.gamesys.corp/10.194.194.147:60268 failed at hudson.remoting.Channel.call(Channel.java:917) at hudson.Launcher$RemoteLauncher.launch(Launcher.java:1053) at hudson.Launcher$ProcStarter.start(Launcher.java:450) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:109) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744) at hudson.model.Build$BuildExecution.build(Build.java:206) at hudson.model.Build$BuildExecution.doRun(Build.java:163) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504) at hudson.model.Run.execute(Run.java:1727) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429) Build step 'Execute shell' marked build as failure
Starting the agent up:
Jan 02, 2018 7:55:08 PM hudson.remoting.jnlp.Main createEngine INFO: Setting up slave: <SlaveName> Jan 02, 2018 7:55:08 PM hudson.remoting.jnlp.Main$CuiListener <init> INFO: Jenkins agent is running in headless mode. Jan 02, 2018 7:55:08 PM hudson.remoting.Engine startEngine WARNING: No Working Directory. Using the legacy JAR Cache location: /Users/mdbuildserver/.jenkins/cache/jars Jan 02, 2018 7:55:08 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Locating server among [https://<jenkins URL>/] Jan 02, 2018 7:55:08 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping] Jan 02, 2018 7:55:08 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Agent discovery successful Agent address: <jenkins URL>/ Agent port: 42650 Identity: fb:f3:24:67:05:be:94:06:49:2a:aa:64:c2:83:32:c7 Jan 02, 2018 7:55:08 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Handshaking Jan 02, 2018 7:55:08 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Connecting to nps-jenkins.gamesys.co.uk:42650 Jan 02, 2018 7:55:08 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Trying protocol: JNLP4-connect Jan 02, 2018 7:55:08 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Remote identity confirmed: fb:f3:24:67:05:be:94:06:49:2a:aa:64:c2:83:32:c7 Jan 02, 2018 7:55:08 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Connected Jan 02, 2018 7:55:08 PM hudson.remoting.UserRequest perform WARNING: LinkageError while performing UserRequest:hudson.slaves.ChannelPinger$SetUpRemotePing@2905 java.lang.NoSuchMethodError: hudson.slaves.ChannelPinger$SetUpRemotePing.getOpenChannelOrFail()Lhudson/remoting/Channel; at hudson.slaves.ChannelPinger$SetUpRemotePing.call(ChannelPinger.java:137) at hudson.slaves.ChannelPinger$SetUpRemotePing.call(ChannelPinger.java:121) at hudson.remoting.UserRequest.perform(UserRequest.java:207) at hudson.remoting.UserRequest.perform(UserRequest.java:53) at hudson.remoting.Request$2.run(Request.java:358) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at hudson.remoting.Engine$1$1.run(Engine.java:98) at java.lang.Thread.run(Thread.java:748) Jan 02, 2018 7:55:08 PM hudson.remoting.UserRequest perform WARNING: LinkageError while performing UserRequest:jenkins.slaves.StandardOutputSwapper$ChannelSwapper@3b16348f java.lang.NoSuchMethodError: jenkins.slaves.StandardOutputSwapper$ChannelSwapper.getOpenChannelOrFail()Lhudson/remoting/Channel; at jenkins.slaves.StandardOutputSwapper$ChannelSwapper.call(StandardOutputSwapper.java:42) at jenkins.slaves.StandardOutputSwapper$ChannelSwapper.call(StandardOutputSwapper.java:39) at hudson.remoting.UserRequest.perform(UserRequest.java:207) at hudson.remoting.UserRequest.perform(UserRequest.java:53) at hudson.remoting.Request$2.run(Request.java:358) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at hudson.remoting.Engine$1$1.run(Engine.java:98) at java.lang.Thread.run(Thread.java:748) Jan 02, 2018 7:55:09 PM com.youdevise.hudson.slavestatus.SlaveListener call INFO: Slave-status listener starting Jan 02, 2018 7:55:09 PM com.youdevise.hudson.slavestatus.SocketHTTPListener waitForConnection INFO: Slave-status listener ready on port 3141
Agent is launched like so:
java -jar agent.jar -jnlpUrl https://<Jenkins URL>/computer/<AgentName>/slave-agent.jnlp -secret <Agent Secret>
panajev could you please create a separate ticket for now? I am not 100% sure the issues are same, but in the case of your stacktraces I know what to fix
Regarding the issue from panajev, updating agent to Remoting 3.15 is a workaround, working on a fix
Thanks oleg_nenashev if you want me to create a separate ticket I will do so right away and link it here.
There is a chance that my patch for JENKINS-48761 solves this issue as well
Most likely it is fixed in Jenkins 2.100, the release is in progress. Retrospective thread: https://groups.google.com/forum/#!topic/jenkinsci-dev/-VQ5YU0yYhg
Code changed in jenkins
User: Oleg Nenashev
Path:
content/_data/changelogs/weekly.yml
http://jenkins-ci.org/commit/jenkins.io/ac42d83bf51e56e826991839a5471eccdd767543
Log:
Changelog: noting JENKINS-48761/JENKINS-48754 in Jenkins 2.100.
The PR also adds warning banners to 2.98/2.99
Code changed in jenkins
User: R. Tyler Croy
Path:
content/_data/changelogs/weekly.yml
http://jenkins-ci.org/commit/jenkins.io/99fe6d097fd1cc1f6797ad7b816c98dec3261a37
Log:
Merge pull request #1305 from oleg-nenashev/changelog/2.100
Changelog: noting JENKINS-48761/JENKINS-48754 in Jenkins 2.100.
Compare: https://github.com/jenkins-infra/jenkins.io/compare/62362762ff55...99fe6d097fd1
Hi,
Thank you for your actions. I have updated to the 2.100 version, restarted Jenkins, then go to my slave and refreshed my browser cache, finally I launch again the agent from the browser and unfortunally the result was identical. It doesn't work. I also tried to download the agent.jar and launch by command line, we have exactly the same thing. Same stack trace on the agent side...
I edited the downloaded agent.jar, it was still in 3.5 version, is it normal ?
However we are totally blocked without the possibility to come back to the 2.94 version (we can only come back to 2.99), do you think we have a possibility to come back to this version through the web interface (because we have no access to the docker administration where Jenkins is installed).
Thank you in advance for your help.
Regards,
Bruno.
Thanks for the quick reply, digging in.
> I edited the downloaded agent.jar, it was still in 3.5 version, is it normal ?
It's not. The version should be 3.15, maybe it's just a typo. But yes, all changes in 2.100 were on the master side, Remoting has not been changed.
> However we are totally blocked without the possibility to come back to the 2.94 version (we can only come back to 2.99), do you think we have a possibility to come back to this version through the web interface (because we have no access to the docker administration where Jenkins is installed).
I am not sure the version rollback from Jenkins UI operates correctly in Docker.
My recommendation would be to replace a container.
Thanks for the pointers regarding Docker though. Do you use the standard Jenkins image or your custom one?
For the version 3.5 it's my bad, sorry, it's the 3.15 (bad typing).
EDIT: For the Docker image, I dont know sorry, it's not me who managed it. I can ask to another person.
OK, Docker... So far I can say the following:
- Master runs on the 8082 port according to the bundle, but the web UI suggests the 10004 port. It is a whatever port mapping or external proxy, I'd guess
- Agent fails to connect to TcpAgentListener port, the connection is refused. Probably it's just because the port is not exposed
- TcpAgentListener runs with port set to 0. It means the it will be taking a random port every restart
- Since you run the master in Docker, I am not sure how it is supposed to work. You would have to expose each port on the Docker host. I doubt you do that.
Please check which port is exposed up for TcpAgentListener, and make sure the same port is configured in the security global config. Since you run in Docker, I would definitely recommend setting port flags so that the port won't be changeable from UI:
-Djenkins.model.Jenkins.slaveAgentPort=${YOUR_EXPOSED_PORT} -Djenkins.model.Jenkins.slaveAgentPortEnforce=true
"TCP agent listener port=0" id=86 (0x56) state=RUNNABLE cpu=0% (running in native) at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) at hudson.TcpSlaveAgentListener.run(TcpSlaveAgentListener.java:157)
Yes it's ok for that, moreover the configuration has not changed on that. We have put the 8081 port for Tcp agents with only JNLP/4 and it is accessible and configured on our Docker. (It was working well before updates).
Back to this story...
> We have put the 8081 port
How did you do that? The threaddump definitely suggests you're running with a random port
Hi,
Sorry for late answer. But you are right, the configuration of the connection through tuneling was not set for the slave. I forgot it after my second installation (it was set only in the global security part).
So after correcting this part the new version has worked perfectly.
Thank you for your support.
Regards,
Bruno.
Thanks for the reply! So this is not a defect. I will close it as a duplicate of JENKINS-48761, because the most of the voters/watchers followed the summary of the issue && timing was very coincidental
Hi,
I've exactly the same problem.
Agent log :
janv. 02, 2018 3:05:49 PM hudson.remoting.jnlp.Main createEngine
INFO: Setting up slave: Windows Slave
janv. 02, 2018 3:05:49 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using D:\Jenkins\remoting as a remoting work directory
Both error and output logs will be printed to D:\Jenkins\remoting
janv. 02, 2018 3:05:49 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
janv. 02, 2018 3:06:00 PM hudson.remoting.jnlp.GuiListener$1 run
INFO: Connecting to ciserver:8081 (retrying:2)
java.io.IOException: Failed to connect to ciserver:8081
{{ at org.jenkinsci.remoting.engine.JnlpAgentEndpoint.open(JnlpAgentEndpoint.java:242)}}
{{ at hudson.remoting.Engine.connect(Engine.java:686)}}
{{ at hudson.remoting.Engine.innerRun(Engine.java:547)}}
{{ at hudson.remoting.Engine.run(Engine.java:469)}}
Caused by: java.net.ConnectException: Connection refused: connect
{{ at sun.nio.ch.Net.connect0(Native Method)}}
{{ at sun.nio.ch.Net.connect(Unknown Source)}}
{{ at sun.nio.ch.Net.connect(Unknown Source)}}
{{ at sun.nio.ch.SocketChannelImpl.connect(Unknown Source)}}
{{ at java.nio.channels.SocketChannel.open(Unknown Source)}}
{{ at org.jenkinsci.remoting.engine.JnlpAgentEndpoint.open(JnlpAgentEndpoint.java:203)}}
{{ ... 3 more}}
And in master log nothing appears...
Regards,
Bruno.