[JENKINS-67732] How to enable DEBUG and TRACE log for Jenkins Master pod and Agent pod

Type: Bug
Resolution: Unresolved
Priority: Blocker
Component/s: kubernetes-client-api-plugin
Labels:
- jenkins
Environment:
Prod

Similar Issues:
Powered by SuggestiMate

Show
Sprint:
Blue Ocean - Candidates
Released As:
~~JENKINS-67664~~

How can i enable Trace log for my Master pod and agent pod.

Few of my Jenkins agent pods are failing and getting this Error, But its not enough to troubleshoot need more debug log to get idea about this Error:

Feb 03, 2022 10:12:14 PM hudson.remoting.RemoteInvocationHandler$Unexporter reportStats
INFO: rate(1min) = 61.4±56.7/sec; rate(5min) = 104.3±40.0/sec; rate(15min) = 114.9±25.1/sec; rate(total) = 23.4±41.1/sec; N = 11
Feb 03, 2022 10:13:14 PM hudson.remoting.RemoteInvocationHandler$Unexporter reportStats
INFO: rate(1min) = 22.6±45.4/sec; rate(5min) = 85.4±54.1/sec; rate(15min) = 107.5±37.2/sec; rate(total) = 11.2±30.7/sec; N = 23
Feb 03, 2022 10:14:14 PM hudson.remoting.RemoteInvocationHandler$Unexporter reportStats
INFO: rate(1min) = 8.3±29.6/sec; rate(5min) = 69.9±59.0/sec; rate(15min) = 100.6±44.6/sec; rate(total) = 7.4±25.5/sec; N = 35
Feb 03, 2022 10:16:03 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Terminated
Feb 03, 2022 10:16:13 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Performing onReconnect operation.
Feb 03, 2022 10:16:13 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: onReconnect operation failed.
Feb 03, 2022 10:16:13 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among [http://jenkins.jenkins.svc.cluster.local:8080/]
Feb 03, 2022 10:16:13 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
Feb 03, 2022 10:16:13 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting TCP connection tunneling is enabled. Skipping the TCP Agent Listener Port availability check
Feb 03, 2022 10:16:13 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Agent discovery successful
  Agent address: jenkins-agent.jenkins.svc.cluster.local
  Agent port:    50000
  Identity:      51:e2:27:2d:d9:02:58:31:e6:22:74:e0:95:2e:a9:4b
Feb 03, 2022 10:16:13 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Feb 03, 2022 10:16:13 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to jenkins-agent.jenkins.svc.cluster.local:50000
Feb 03, 2022 10:16:13 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP4-connect
Feb 03, 2022 10:16:13 PM org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader run
INFO: Waiting for ProtocolStack to start.
Feb 03, 2022 10:16:13 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Remote identity confirmed: 51:e2:27:2d:d9:02:58:31:e6:22:74:e0:95:2e:a9:4b
Feb 03, 2022 10:16:13 PM org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer onRecv
INFO: [JNLP4-connect connection to jenkins-agent.jenkins.svc.cluster.local/10.29.15.114:50000] Local headers refused by remote: Unknown client name: tsds-tsds-api-main-71-wn5bz-ldf7h-6kkt4
Feb 03, 2022 10:16:13 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Protocol JNLP4-connect encountered an unexpected exception
java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Unknown client name: tsds-tsds-api-main-71-wn5bz-ldf7h-6kkt4
	at org.jenkinsci.remoting.util.SettableFuture.get(SettableFuture.java:223)
	at hudson.remoting.Engine.innerRun(Engine.java:778)
	at hudson.remoting.Engine.run(Engine.java:540)
Caused by: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Unknown client name: tsds-tsds-api-main-71-wn5bz-ldf7h-6kkt4
	at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.newAbortCause(ConnectionHeadersFilterLayer.java:378)
	at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.onRecvClosed(ConnectionHeadersFilterLayer.java:433)
	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:825)
	at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:288)
	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:170)
	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:825)
	at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154)
	at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer.access$1500(BIONetworkLayer.java:49)
	at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader.run(BIONetworkLayer.java:255)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:122)
	at java.base/java.lang.Thread.run(Unknown Source)
	Suppressed: java.nio.channels.ClosedChannelException
		... 7 more


Feb 03, 2022 10:16:13 PM hudson.remoting.jnlp.Main$CuiListener error
SEVERE: The server rejected the connection: None of the protocols were accepted
java.lang.Exception: The server rejected the connection: None of the protocols were accepted
	at hudson.remoting.Engine.onConnectionRejected(Engine.java:864)
	at hudson.remoting.Engine.innerRun(Engine.java:804)
	at hudson.remoting.Engine.run(Engine.java:540)

SparkC created issue - 2022-02-03 15:49

SparkC made changes - 2022-02-03 16:00

Assignee

New: SparkC [ rohithg534 ]

SparkC made changes - 2022-02-03 16:00

Assignee

Original: SparkC [ rohithg534 ]

SparkC made changes - 2022-02-03 21:30

Component/s		New: kubernetes-client-api-plugin [ 25921 ]
Component/s	Original: kubernetes-plugin [ 20639 ]

SparkC made changes - 2022-02-03 21:31

Sprint	Original: Bundles management UI Sprint [ 1 ]	New: Blue Ocean - Candidates [ 441 ]
Assignee		New: Kyle Cronin [ kylecronin ]

SparkC made changes - 2022-02-03 22:46

Description

Original: How can i enable Trace log for my Master pod and agent pod.

New: How can i enable Trace log for my Master pod and agent pod.

Few of my Jenkins agent pods are failing and getting this Error, But its not enough to troubleshoot need more debug log to get idea about this Error:

{code:java}
Feb 03, 2022 10:12:14 PM hudson.remoting.RemoteInvocationHandler$Unexporter reportStats
INFO: rate(1min) = 61.4±56.7/sec; rate(5min) = 104.3±40.0/sec; rate(15min) = 114.9±25.1/sec; rate(total) = 23.4±41.1/sec; N = 11
Feb 03, 2022 10:13:14 PM hudson.remoting.RemoteInvocationHandler$Unexporter reportStats
INFO: rate(1min) = 22.6±45.4/sec; rate(5min) = 85.4±54.1/sec; rate(15min) = 107.5±37.2/sec; rate(total) = 11.2±30.7/sec; N = 23
Feb 03, 2022 10:14:14 PM hudson.remoting.RemoteInvocationHandler$Unexporter reportStats
INFO: rate(1min) = 8.3±29.6/sec; rate(5min) = 69.9±59.0/sec; rate(15min) = 100.6±44.6/sec; rate(total) = 7.4±25.5/sec; N = 35
Feb 03, 2022 10:16:03 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Terminated
Feb 03, 2022 10:16:13 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Performing onReconnect operation.
Feb 03, 2022 10:16:13 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: onReconnect operation failed.
Feb 03, 2022 10:16:13 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among [http://jenkins.jenkins.svc.cluster.local:8080/]
Feb 03, 2022 10:16:13 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
Feb 03, 2022 10:16:13 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting TCP connection tunneling is enabled. Skipping the TCP Agent Listener Port availability check
Feb 03, 2022 10:16:13 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Agent discovery successful
Agent address: jenkins-agent.jenkins.svc.cluster.local
Agent port: 50000
Identity: 51:e2:27:2d:d9:02:58:31:e6:22:74:e0:95:2e:a9:4b
Feb 03, 2022 10:16:13 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Feb 03, 2022 10:16:13 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to jenkins-agent.jenkins.svc.cluster.local:50000
Feb 03, 2022 10:16:13 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP4-connect
Feb 03, 2022 10:16:13 PM org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader run
INFO: Waiting for ProtocolStack to start.
Feb 03, 2022 10:16:13 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Remote identity confirmed: 51:e2:27:2d:d9:02:58:31:e6:22:74:e0:95:2e:a9:4b
Feb 03, 2022 10:16:13 PM org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer onRecv
INFO: [JNLP4-connect connection to jenkins-agent.jenkins.svc.cluster.local/10.29.15.114:50000] Local headers refused by remote: Unknown client name: tsds-tsds-api-main-71-wn5bz-ldf7h-6kkt4
Feb 03, 2022 10:16:13 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Protocol JNLP4-connect encountered an unexpected exception
java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Unknown client name: tsds-tsds-api-main-71-wn5bz-ldf7h-6kkt4
at org.jenkinsci.remoting.util.SettableFuture.get(SettableFuture.java:223)
at hudson.remoting.Engine.innerRun(Engine.java:778)
at hudson.remoting.Engine.run(Engine.java:540)
Caused by: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Unknown client name: tsds-tsds-api-main-71-wn5bz-ldf7h-6kkt4
at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.newAbortCause(ConnectionHeadersFilterLayer.java:378)
at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.onRecvClosed(ConnectionHeadersFilterLayer.java:433)
at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:825)
at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:288)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:170)
at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:825)
at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154)
at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer.access$1500(BIONetworkLayer.java:49)
at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader.run(BIONetworkLayer.java:255)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:122)
at java.base/java.lang.Thread.run(Unknown Source)
Suppressed: java.nio.channels.ClosedChannelException
... 7 more

Feb 03, 2022 10:16:13 PM hudson.remoting.jnlp.Main$CuiListener error
SEVERE: The server rejected the connection: None of the protocols were accepted
java.lang.Exception: The server rejected the connection: None of the protocols were accepted
at hudson.remoting.Engine.onConnectionRejected(Engine.java:864)
at hudson.remoting.Engine.innerRun(Engine.java:804)
at hudson.remoting.Engine.run(Engine.java:540)

{code}

Benjamin RIch made changes - 2022-02-09 15:54

Assignee

Original: Kyle Cronin [ kylecronin ]

New: Benjamin RIch [ richbg ]

Benjamin RIch made changes - 2022-02-09 15:56

Assignee

Original: Benjamin RIch [ richbg ]

New: Kyle Cronin [ kylecronin ]

Benjamin RIch made changes - 2022-02-09 15:57

Assignee

Original: Kyle Cronin [ kylecronin ]

Assignee:: Unassigned

Reporter:: SparkC

Votes:: 0 Vote for this issue

Watchers:: 1 Start watching this issue

Created:: 2022-02-03 15:49

Updated:: 2022-02-09 15:57

Jenkins

Details

Description

Attachments

Activity

People

Dates