gillesdehaudt this doesn't ring any bell.

I have a couple of questions:

Since some version of Jenkins LTS

Any idea when this started? Is it tied to a Jenkins core update or a plugin update?

we have some old audit-trail log presents

Are they in the JENKINS_HOME or somewhere else?

Jenkins will take more than 20 minutes to boot

Could you please reproduce the issue, and while the boot is going on, take several threaddumps so that we can understand better what's going on? As this is a public tracker, make sure to review those threaddump and offuscate any information that you don't want to divulge.

In addition, would you mind sharing the Core version of Jenkins as well as the Audit Trail plugin version you are using?

Thanks for taking the time to  look at my problem.

Here the answer that I can quickly get

We discovered the problems with the 2.222.1 versions.

The audit trail log are stored in the JENKINS_HOME directory

Currently we are using audit trail plugin version 3.5 and Jenkins 2.222.4

I'm building an environment to reproduce the problem and I will get back to you as soon as I've got the requested information.

threaddump.out

As requested, I've uploaded one threaddump of jenkins start when we encounter the problem.

If it may help you, I noticed that I have a lot of audit-trail.log.0.*.lck created. One for each start of jenkins I think.

// code placeholder
ls logs/audit-trail.log.0*
logs/audit-trail.log.0         logs/audit-trail.log.0.13      logs/audit-trail.log.0.17      logs/audit-trail.log.0.20      logs/audit-trail.log.0.24      logs/audit-trail.log.0.28      logs/audit-trail.log.0.4.lck  logs/audit-trail.log.0.8.lck
logs/audit-trail.log.0.1       logs/audit-trail.log.0.13.lck  logs/audit-trail.log.0.17.lck  logs/audit-trail.log.0.20.lck  logs/audit-trail.log.0.24.lck  logs/audit-trail.log.0.28.lck  logs/audit-trail.log.0.5      logs/audit-trail.log.0.9
logs/audit-trail.log.0.10      logs/audit-trail.log.0.14      logs/audit-trail.log.0.18      logs/audit-trail.log.0.21      logs/audit-trail.log.0.25      logs/audit-trail.log.0.29      logs/audit-trail.log.0.5.lck  logs/audit-trail.log.0.9.lck
logs/audit-trail.log.0.10.lck  logs/audit-trail.log.0.14.lck  logs/audit-trail.log.0.18.lck  logs/audit-trail.log.0.21.lck  logs/audit-trail.log.0.25.lck  logs/audit-trail.log.0.29.lck  logs/audit-trail.log.0.6      logs/audit-trail.log.0.lck
logs/audit-trail.log.0.11      logs/audit-trail.log.0.15      logs/audit-trail.log.0.19      logs/audit-trail.log.0.22      logs/audit-trail.log.0.26      logs/audit-trail.log.0.2.lck   logs/audit-trail.log.0.6.lck
logs/audit-trail.log.0.11.lck  logs/audit-trail.log.0.15.lck  logs/audit-trail.log.0.19.lck  logs/audit-trail.log.0.22.lck  logs/audit-trail.log.0.26.lck  logs/audit-trail.log.0.3       logs/audit-trail.log.0.7
logs/audit-trail.log.0.12      logs/audit-trail.log.0.16      logs/audit-trail.log.0.1.lck   logs/audit-trail.log.0.23      logs/audit-trail.log.0.27      logs/audit-trail.log.0.3.lck   logs/audit-trail.log.0.7.lck
logs/audit-trail.log.0.12.lck  logs/audit-trail.log.0.16.lck  logs/audit-trail.log.0.2       logs/audit-trail.log.0.23.lck  logs/audit-trail.log.0.27.lck  logs/audit-trail.log.0.4       logs/audit-trail.log.0.8

gillesdehaudt 

Nothing looks suspicious in the thread dump. Would you mind sharing the configuration of the plugin (a screenshot would be enough, you can hide the paths and server info if any, what I'm really interested in is the Log File Size MB and Log File Count settings). Additionally, could you please give me the output of ls -al logs I would like to get a better idea of the size of the log files you have.

Thanks

Sorry for my late response

Here is the audittrail config file : 

cat audit-trail.xml
<?xml version='1.1' encoding='UTF-8'?>
<hudson.plugins.audit__trail.AuditTrailPlugin plugin="audit-trail@3.5">
<pattern>.*/(?:login|job|console|configSubmit|doDelete|postBuildResult|enable|disable|cancelQueue|stop|toggleLogKeep|doWipeOutWorkspace|createItem|createView|toggleOffline|cancelQuietDown|quietDown|restart|exit|safeExit)</pattern>
<logBuildCause>true</logBuildCause>
<loggers>
<hudson.plugins.audit__trail.LogFileAuditLogger>
<logSeparator> </logSeparator>
<log>/var/jenkins_home/logs/audit-trail.log</log>
<limit>1</limit>
<count>5</count>
</hudson.plugins.audit__trail.LogFileAuditLogger>
</loggers>
</hudson.plugins.audit__trail.AuditTrailPlugin>

and here is the ls from logs directory : 

ls -la logs.bck
total 3856
drwxr-xr-x 5 root root 5611 Jun 17 04:04 .
drwxr-xr-x 24 root root 3082 Jul 8 14:39 ..
rw-rr- 1 root root 0 Jun 15 10:05 audit-trail.log.0
rw-rr- 1 root root 37045 Jun 15 10:56 audit-trail.log.0.1
rw-rr- 1 root root 50 Jun 15 11:19 audit-trail.log.0.10
rw-rr- 1 root root 0 Jun 15 11:16 audit-trail.log.0.10.lck
rw-rr- 1 root root 0 Jun 15 11:21 audit-trail.log.0.11
rw-rr- 1 root root 0 Jun 15 11:21 audit-trail.log.0.11.lck
rw-rr- 1 root root 0 Jun 15 11:23 audit-trail.log.0.12
rw-rr- 1 root root 0 Jun 15 11:22 audit-trail.log.0.12.lck
rw-rr- 1 root root 5998 Jun 15 15:06 audit-trail.log.0.13
rw-rr- 1 root root 0 Jun 15 11:24 audit-trail.log.0.13.lck
rw-rr- 1 root root 0 Jun 15 15:07 audit-trail.log.0.14
rw-rr- 1 root root 0 Jun 15 15:07 audit-trail.log.0.14.lck
rw-rr- 1 root root 0 Jun 15 15:08 audit-trail.log.0.15
rw-rr- 1 root root 0 Jun 15 15:08 audit-trail.log.0.15.lck
rw-rr- 1 root root 31030 Jun 15 16:03 audit-trail.log.0.16
rw-rr- 1 root root 0 Jun 15 15:10 audit-trail.log.0.16.lck
rw-rr- 1 root root 0 Jun 15 16:06 audit-trail.log.0.17
rw-rr- 1 root root 0 Jun 15 16:06 audit-trail.log.0.17.lck
rw-rr- 1 root root 0 Jun 15 16:08 audit-trail.log.0.18
rw-rr- 1 root root 0 Jun 15 16:08 audit-trail.log.0.18.lck
rw-rr- 1 root root 3948 Jun 15 16:17 audit-trail.log.0.19
rw-rr- 1 root root 0 Jun 15 16:09 audit-trail.log.0.19.lck
rw-rr- 1 root root 0 Jun 15 10:05 audit-trail.log.0.1.lck
rw-rr- 1 root root 0 Jun 15 10:58 audit-trail.log.0.2
rw-rr- 1 root root 0 Jun 15 16:19 audit-trail.log.0.20
rw-rr- 1 root root 0 Jun 15 16:19 audit-trail.log.0.20.lck
rw-rr- 1 root root 0 Jun 15 16:21 audit-trail.log.0.21
rw-rr- 1 root root 0 Jun 15 16:21 audit-trail.log.0.21.lck
rw-rr- 1 root root 0 Jun 15 16:24 audit-trail.log.0.22
rw-rr- 1 root root 0 Jun 15 16:24 audit-trail.log.0.22.lck
rw-rr- 1 root root 0 Jun 15 16:27 audit-trail.log.0.23
rw-rr- 1 root root 0 Jun 15 16:27 audit-trail.log.0.23.lck
rw-rr- 1 root root 0 Jun 15 16:29 audit-trail.log.0.24
rw-rr- 1 root root 0 Jun 15 16:29 audit-trail.log.0.24.lck
rw-rr- 1 root root 0 Jun 15 16:34 audit-trail.log.0.25
rw-rr- 1 root root 0 Jun 15 16:34 audit-trail.log.0.25.lck
rw-rr- 1 root root 0 Jun 15 16:37 audit-trail.log.0.26
rw-rr- 1 root root 0 Jun 15 16:37 audit-trail.log.0.26.lck
rw-rr- 1 root root 0 Jun 15 16:42 audit-trail.log.0.27
rw-rr- 1 root root 0 Jun 15 16:42 audit-trail.log.0.27.lck
rw-rr- 1 root root 0 Jun 15 16:45 audit-trail.log.0.28
rw-rr- 1 root root 0 Jun 15 16:45 audit-trail.log.0.28.lck
rw-rr- 1 root root 0 Jun 15 16:50 audit-trail.log.0.29
rw-rr- 1 root root 0 Jun 15 16:50 audit-trail.log.0.29.lck
rw-rr- 1 root root 0 Jun 15 10:58 audit-trail.log.0.2.lck
rw-rr- 1 root root 0 Jun 15 10:59 audit-trail.log.0.3
rw-rr- 1 root root 0 Jun 15 16:53 audit-trail.log.0.30
rw-rr- 1 root root 0 Jun 15 16:53 audit-trail.log.0.30.lck
rw-rr- 1 root root 0 Jun 15 16:57 audit-trail.log.0.31

rw-rr- 1 root root 0 Jun 15 16:57 audit-trail.log.0.31.lck
rw-rr- 1 root root 0 Jun 15 17:01 audit-trail.log.0.32
rw-rr- 1 root root 0 Jun 15 17:01 audit-trail.log.0.32.lck
rw-rr- 1 root root 0 Jun 15 17:05 audit-trail.log.0.33
rw-rr- 1 root root 0 Jun 15 17:05 audit-trail.log.0.33.lck
rw-rr- 1 root root 0 Jun 15 17:12 audit-trail.log.0.34
rw-rr- 1 root root 0 Jun 15 17:12 audit-trail.log.0.34.lck
rw-rr- 1 root root 0 Jun 15 17:19 audit-trail.log.0.35
rw-rr- 1 root root 0 Jun 15 17:19 audit-trail.log.0.35.lck
rw-rr- 1 root root 0 Jun 15 17:27 audit-trail.log.0.36
rw-rr- 1 root root 0 Jun 15 17:27 audit-trail.log.0.36.lck
rw-rr- 1 root root 0 Jun 15 17:34 audit-trail.log.0.37
rw-rr- 1 root root 0 Jun 15 17:34 audit-trail.log.0.37.lck
rw-rr- 1 root root 0 Jun 15 17:41 audit-trail.log.0.38
rw-rr- 1 root root 0 Jun 15 17:41 audit-trail.log.0.38.lck
rw-rr- 1 root root 0 Jun 15 17:49 audit-trail.log.0.39
rw-rr- 1 root root 0 Jun 15 17:49 audit-trail.log.0.39.lck
rw-rr- 1 root root 0 Jun 15 10:59 audit-trail.log.0.3.lck
rw-rr- 1 root root 281 Jun 15 11:06 audit-trail.log.0.4
rw-rr- 1 root root 0 Jun 15 17:56 audit-trail.log.0.40
rw-rr- 1 root root 0 Jun 15 17:56 audit-trail.log.0.40.lck
rw-rr- 1 root root 0 Jun 15 18:03 audit-trail.log.0.41
rw-rr- 1 root root 0 Jun 15 18:03 audit-trail.log.0.41.lck
rw-rr- 1 root root 0 Jun 15 18:11 audit-trail.log.0.42
rw-rr- 1 root root 0 Jun 15 18:11 audit-trail.log.0.42.lck
rw-rr- 1 root root 0 Jun 15 18:18 audit-trail.log.0.43
rw-rr- 1 root root 0 Jun 15 18:18 audit-trail.log.0.43.lck
rw-rr- 1 root root 0 Jun 15 18:26 audit-trail.log.0.44
rw-rr- 1 root root 0 Jun 15 18:26 audit-trail.log.0.44.lck
rw-rr- 1 root root 0 Jun 15 18:33 audit-trail.log.0.45
rw-rr- 1 root root 0 Jun 15 18:33 audit-trail.log.0.45.lck
rw-rr- 1 root root 0 Jun 15 18:40 audit-trail.log.0.46
rw-rr- 1 root root 0 Jun 15 18:40 audit-trail.log.0.46.lck
rw-rr- 1 root root 0 Jun 15 18:48 audit-trail.log.0.47
rw-rr- 1 root root 0 Jun 15 18:48 audit-trail.log.0.47.lck
rw-rr- 1 root root 0 Jun 15 18:56 audit-trail.log.0.48
rw-rr- 1 root root 0 Jun 15 18:55 audit-trail.log.0.48.lck
rw-rr- 1 root root 0 Jun 15 19:03 audit-trail.log.0.49
rw-rr- 1 root root 0 Jun 15 19:03 audit-trail.log.0.49.lck
rw-rr- 1 root root 0 Jun 15 10:59 audit-trail.log.0.4.lck
rw-rr- 1 root root 0 Jun 15 11:05 audit-trail.log.0.5
rw-rr- 1 root root 0 Jun 15 19:10 audit-trail.log.0.50
rw-rr- 1 root root 0 Jun 15 19:10 audit-trail.log.0.50.lck
rw-rr- 1 root root 0 Jun 15 19:18 audit-trail.log.0.51
rw-rr- 1 root root 0 Jun 15 19:17 audit-trail.log.0.51.lck
rw-rr- 1 root root 0 Jun 15 19:25 audit-trail.log.0.52
rw-rr- 1 root root 0 Jun 15 19:25 audit-trail.log.0.52.lck
rw-rr- 1 root root 0 Jun 15 19:32 audit-trail.log.0.53
rw-rr- 1 root root 0 Jun 15 19:32 audit-trail.log.0.53.lck
rw-rr- 1 root root 0 Jun 15 19:40 audit-trail.log.0.54
rw-rr- 1 root root 0 Jun 15 19:39 audit-trail.log.0.54.lck
rw-rr- 1 root root 0 Jun 15 19:47 audit-trail.log.0.55
rw-rr- 1 root root 0 Jun 15 19:47 audit-trail.log.0.55.lck
rw-rr- 1 root root 0 Jun 15 19:54 audit-trail.log.0.56
rw-rr- 1 root root 0 Jun 15 19:54 audit-trail.log.0.56.lck
rw-rr- 1 root root 0 Jun 15 20:02 audit-trail.log.0.57
rw-rr- 1 root root 0 Jun 15 20:01 audit-trail.log.0.57.lck
rw-rr- 1 root root 0 Jun 15 20:09 audit-trail.log.0.58
rw-rr- 1 root root 0 Jun 15 20:09 audit-trail.log.0.58.lck
rw-rr- 1 root root 0 Jun 15 20:17 audit-trail.log.0.59
rw-rr- 1 root root 0 Jun 15 20:17 audit-trail.log.0.59.lck
rw-rr- 1 root root 0 Jun 15 11:05 audit-trail.log.0.5.lck
rw-rr- 1 root root 0 Jun 15 11:05 audit-trail.log.0.6
rw-rr- 1 root root 0 Jun 15 20:24 audit-trail.log.0.60
rw-rr- 1 root root 0 Jun 15 20:24 audit-trail.log.0.60.lck
rw-rr- 1 root root 0 Jun 15 20:32 audit-trail.log.0.61
rw-rr- 1 root root 0 Jun 15 20:32 audit-trail.log.0.61.lck
rw-rr- 1 root root 0 Jun 15 20:40 audit-trail.log.0.62
rw-rr- 1 root root 0 Jun 15 20:40 audit-trail.log.0.62.lck

rw-rr- 1 root root 0 Jun 15 20:47 audit-trail.log.0.63
rw-rr- 1 root root 0 Jun 15 20:47 audit-trail.log.0.63.lck
rw-rr- 1 root root 0 Jun 15 20:55 audit-trail.log.0.64
rw-rr- 1 root root 0 Jun 15 20:55 audit-trail.log.0.64.lck
rw-rr- 1 root root 0 Jun 15 21:02 audit-trail.log.0.65
rw-rr- 1 root root 0 Jun 15 21:02 audit-trail.log.0.65.lck
rw-rr- 1 root root 0 Jun 15 21:10 audit-trail.log.0.66
rw-rr- 1 root root 0 Jun 15 21:10 audit-trail.log.0.66.lck
rw-rr- 1 root root 0 Jun 15 21:17 audit-trail.log.0.67
rw-rr- 1 root root 0 Jun 15 21:17 audit-trail.log.0.67.lck
rw-rr- 1 root root 0 Jun 15 21:25 audit-trail.log.0.68.lck
rw-rr- 1 root root 0 Jun 17 04:04 audit-trail.log.0.69.lck
rw-rr- 1 root root 0 Jun 15 11:05 audit-trail.log.0.6.lck
rw-rr- 1 root root 3792 Jun 15 11:12 audit-trail.log.0.7
rw-rr- 1 root root 0 Jun 15 11:06 audit-trail.log.0.7.lck
rw-rr- 1 root root 0 Jun 15 11:14 audit-trail.log.0.8
rw-rr- 1 root root 0 Jun 15 11:14 audit-trail.log.0.8.lck
rw-rr- 1 root root 0 Jun 15 11:15 audit-trail.log.0.9
rw-rr- 1 root root 0 Jun 15 11:15 audit-trail.log.0.9.lck
rw-rr- 1 root root 0 Jun 15 10:05 audit-trail.log.0.lck
rw-rr- 1 root root 2851 Jun 17 10:55 health-checker.log
drwxr-xr-x 2 root root 0 Jun 15 16:31 slaves
drwxr-xr-x 2 root root 0 Jun 15 10:05 sse-events
drwxr-xr-x 2 root root 1065 Jun 17 12:34 tasks

Gilles

gillesdehaudt

I cannot figure out what's going on. What's for sure is that the number of lck file is not normal, I would expect to see 3 of them (it's a known issue in the plugin, the file handlers is initialised 3 times, but it's generally not an issue as the same lck file is always used). What's really strange about your use case is the number of lock files, I have no clue how you end up in this situation of having that many lck files created at the same time. I did some tests and could not reproduce the issue...

I believe the next step is to understand exactly what the JVM is doing during the startup. For this, we will need more than one thread dump. Could you please restart the instance, then take a thread dump each 30s during 5 minutes? This way I hope to be able to see blocked calls that would lead me to an explanation.

Thanks!

gillesdehaudt any update?

Hello,

Sorry for the delay, I think I won't have the time to work on this. We have a workaround (adding a step to remove auditlog file at startup) to avoid problems. I will close the issue and open a new one if needed.

Thanks for your time;

Gilles