[JENKINS-28286] jenkins.log growing extremely fast

Type: Bug
Resolution: Cannot Reproduce
Priority: Major
Component/s: core
Labels:
None
Environment:
OS: Ubuntu 14.04.2 LTS
JDK: OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode)
JRE: IcedTea 2.5.5
Java: 1.7.0_79

Similar Issues:
Powered by SuggestiMate

Show

Since Jenkins 1.612, logs are growing super fast mostly because of repeatedly reported exceptions. Most frequent ones:

jenkins.diagnosis.HsErrPidList <init>
WARNING: Failed to list up hs_err_pid files
java.lang.UnsatisfiedLinkError: Can't obtain updateLastError method for class com.sun.jna.Native
        at com.sun.jna.Native.initIDs(Native Method)
        at com.sun.jna.Native.<clinit>(Native.java:139)
        at com.sun.akuma.CLibrary.<clinit>(CLibrary.java:89)
        at com.sun.akuma.JavaVMArguments.resolvePID(JavaVMArguments.java:126)
        at com.sun.akuma.JavaVMArguments.ofLinux(JavaVMArguments.java:114)
        at com.sun.akuma.JavaVMArguments.of(JavaVMArguments.java:104)
        at com.sun.akuma.JavaVMArguments.current(JavaVMArguments.java:92)
        at jenkins.diagnosis.HsErrPidList.<init>(HsErrPidList.java:72)
        at jenkins.diagnosis.HsErrPidList$$FastClassByGuice$$b61c3f44.newInstance(<generated>)
        at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
        at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:61)
        at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:108)
        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:88)
        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:269)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1058)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
        at com.google.inject.Scopes$1$1.get(Scopes.java:65)
        at hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1.get(ExtensionFinder.java:428)
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
        at com.google.inject.internal.InjectorImpl$3$1.call(InjectorImpl.java:1005)
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1051)
        at com.google.inject.internal.InjectorImpl$3.get(InjectorImpl.java:1001)
        at hudson.ExtensionFinder$GuiceFinder._find(ExtensionFinder.java:390)
        at hudson.ExtensionFinder$GuiceFinder.find(ExtensionFinder.java:381)
        at hudson.ClassicPluginStrategy.findComponents(ClassicPluginStrategy.java:386)
        at hudson.ExtensionList.load(ExtensionList.java:304)
        at hudson.ExtensionList.ensureLoaded(ExtensionList.java:257)
        at hudson.ExtensionList.iterator(ExtensionList.java:146)
        at hudson.ExtensionList.get(ExtensionList.java:137)
        at hudson.diagnosis.NullIdDescriptorMonitor.verifyId(NullIdDescriptorMonitor.java:89)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:106)
        at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:176)
        at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
        at jenkins.model.Jenkins$7.runTask(Jenkins.java:905)
        at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
        at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

hudson.UDPBroadcastThread run
WARNING: UDP handling problem
java.net.SocketException: No such device
        at java.net.PlainDatagramSocketImpl.join(Native Method)
        at java.net.AbstractPlainDatagramSocketImpl.join(AbstractPlainDatagramSocketImpl.java:179)
        at java.net.MulticastSocket.joinGroup(MulticastSocket.java:319)
        at hudson.UDPBroadcastThread.run(UDPBroadcastThread.java:77)

javax.jmdns.impl.tasks.state.DNSStateTask run
WARNING: Prober(fe80:0:0:0:f21f:afff:fe6f:1a20%2.local.).run() exception 
java.io.IOException: Network is unreachable
        at java.net.PlainDatagramSocketImpl.send(Native Method)
        at java.net.DatagramSocket.send(DatagramSocket.java:697)
        at javax.jmdns.impl.JmDNSImpl.send(JmDNSImpl.java:1537)
        at javax.jmdns.impl.tasks.state.DNSStateTask.run(DNSStateTask.java:131)
        at java.util.TimerThread.mainLoop(Timer.java:555)
        at java.util.TimerThread.run(Timer.java:505)

Daniel Beck added a comment - 2015-05-08 09:09

When you write 'since 1.612', does that mean it did not happen in 1.611?

Daniel Beck added a comment - 2015-05-08 09:09 When you write 'since 1.612', does that mean it did not happen in 1.611?

Daniel Beck added a comment - 2015-05-08 09:10

FWIW both UDPBroadcastThread and jmdns can be disabled, https://wiki.jenkins-ci.org/display/JENKINS/Features+controlled+by+system+properties

Daniel Beck added a comment - 2015-05-08 09:10 FWIW both UDPBroadcastThread and jmdns can be disabled, https://wiki.jenkins-ci.org/display/JENKINS/Features+controlled+by+system+properties

Régis Desgroppes added a comment - 2015-05-08 10:04

@DanielBeck thanks for making me double-check: I'm now confused I had close errors with Jenkins 1.611:

"Failed to list up hs_err_pid files" was already present.

already lots of DNS related warnings:

javax.jmdns.impl.DNSIncoming readAnswer
WARNING: There was an OPT answer. Not currently handled. Option code: 65002 data: 2935FB8F05C06E83

=> I will therefore remove the "since Jenkins 1.612" from the title.

Thanks for the tip on disabling some features, but I'm wondering how that would impact Jenkins' behavior.

Am I alone seeing this? We noticed it because of increased I/O times. That said, I don't see anything related in the changelog.

Régis Desgroppes added a comment - 2015-05-08 10:04 @DanielBeck thanks for making me double-check: I'm now confused I had close errors with Jenkins 1.611: "Failed to list up hs_err_pid files" was already present. already lots of DNS related warnings: javax.jmdns.impl.DNSIncoming readAnswer WARNING: There was an OPT answer. Not currently handled. Option code: 65002 data: 2935FB8F05C06E83 => I will therefore remove the "since Jenkins 1.612" from the title. Thanks for the tip on disabling some features, but I'm wondering how that would impact Jenkins' behavior. Am I alone seeing this? We noticed it because of increased I/O times. That said, I don't see anything related in the changelog .

Daniel Beck added a comment - 2015-05-08 11:14

Thanks for the tip on disabling some features, but I'm wondering how that would impact Jenkins' behavior.

mDNS and UDP broadcast are not generally needed, they just announce to your network where Jenkins can be reached.

Would be interesting since when (Jenkins update or OS update) these occur. Did you recently disable IPv4 networking?

What's your CPU architecture?

Daniel Beck added a comment - 2015-05-08 11:14 Thanks for the tip on disabling some features, but I'm wondering how that would impact Jenkins' behavior. mDNS and UDP broadcast are not generally needed, they just announce to your network where Jenkins can be reached. Would be interesting since when (Jenkins update or OS update) these occur. Did you recently disable IPv4 networking? What's your CPU architecture?

Régis Desgroppes added a comment - 2015-05-08 12:00

No major recent OS update (but usual security updates), uptime is 72 days.
IPv4 was never disabled. I even found we had to set -Djava.net.preferIPv4Stack=true a couple of years ago. Nobody remembers why, so I'll revert this.
The machine is a good (5-year) old bi-Xeon exposing 24 virtual CPUs (6-core, hyperthreading).

The more it goes, the more I think I'll cancel this ticket.

Régis Desgroppes added a comment - 2015-05-08 12:00 No major recent OS update (but usual security updates), uptime is 72 days. IPv4 was never disabled. I even found we had to set -Djava.net.preferIPv4Stack=true a couple of years ago. Nobody remembers why, so I'll revert this. The machine is a good (5-year) old bi-Xeon exposing 24 virtual CPUs (6-core, hyperthreading). The more it goes, the more I think I'll cancel this ticket.

Daniel Beck added a comment - 2015-05-08 12:22

To clarify, all of these three exception appear periodically, or at least multiple times per Jenkins start? And not just once when starting Jenkins?

Daniel Beck added a comment - 2015-05-08 12:22 To clarify, all of these three exception appear periodically, or at least multiple times per Jenkins start? And not just once when starting Jenkins?

Régis Desgroppes added a comment - 2015-05-08 12:34

The "WARNING: Failed to list up hs_err_pid files" happens first when Jenkins starts, and then from time to time.
DNS/UDP related warnings don't happen right at start time, but from time to time and flooding the logs.

Régis Desgroppes added a comment - 2015-05-08 12:34 The "WARNING: Failed to list up hs_err_pid files" happens first when Jenkins starts, and then from time to time. DNS/UDP related warnings don't happen right at start time, but from time to time and flooding the logs.

Régis Desgroppes added a comment - 2015-05-18 11:30

Sorry, I should have investigated more before filing this issue. I'll reopen when/if I manage to isolate it.

Régis Desgroppes added a comment - 2015-05-18 11:30 Sorry, I should have investigated more before filing this issue. I'll reopen when/if I manage to isolate it.

Assignee:: Unassigned

Reporter:: Régis Desgroppes

Votes:: 0 Vote for this issue

Watchers:: 1 Start watching this issue

Created:: 2015-05-07 15:11

Updated:: 2015-07-26 18:28

Resolved:: 2015-05-18 11:30

Jenkins

Details

Description

Attachments

Activity

Collapse comment: Daniel Beck added a comment - 2015-05-08 09:09

Expand comment: Daniel Beck added a comment - 2015-05-08 09:09

Collapse comment: Daniel Beck added a comment - 2015-05-08 09:10

Expand comment: Daniel Beck added a comment - 2015-05-08 09:10

Collapse comment: Régis Desgroppes added a comment - 2015-05-08 10:04

Expand comment: Régis Desgroppes added a comment - 2015-05-08 10:04

Collapse comment: Daniel Beck added a comment - 2015-05-08 11:14

Expand comment: Daniel Beck added a comment - 2015-05-08 11:14

Collapse comment: Régis Desgroppes added a comment - 2015-05-08 12:00

Expand comment: Régis Desgroppes added a comment - 2015-05-08 12:00

Collapse comment: Daniel Beck added a comment - 2015-05-08 12:22

Expand comment: Daniel Beck added a comment - 2015-05-08 12:22

Collapse comment: Régis Desgroppes added a comment - 2015-05-08 12:34

Expand comment: Régis Desgroppes added a comment - 2015-05-08 12:34

Collapse comment: Régis Desgroppes added a comment - 2015-05-18 11:30

Expand comment: Régis Desgroppes added a comment - 2015-05-18 11:30

People

Dates