-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
Jenkins 2.289.3
-
Powered by SuggestiMate
Hi,
Jenkins is deployed on EKS.
Everytime Jenkins is redployed, the past jobs are showing dated "1 Jan 1970".
All past jobs are in hanged mode, each time needing to abort from script console.
[JENKINS-66328] pipeline in Jenkins are showing like executed back to 1 Jan 1970
Attachment | New: image-2024-01-25-10-50-09-884.png [ 61994 ] |
I am having the same problem. The builds all look fine on disk, but Jenkins considers them failed and hangs on running a new build until a second restart.
There is no clock problem detectable on the machine.
drwxr-xr-x 7 jenkins jenkins 4096 Dec 13 17:37 204 drwxr-xr-x 4 jenkins jenkins 4096 Jan 5 17:25 205 drwxr-xr-x 6 jenkins jenkins 4096 Jan 5 17:50 206 drwxr-xr-x 6 jenkins jenkins 4096 Jan 23 11:07 207 drwxr-xr-x 6 jenkins jenkins 4096 Jan 25 10:11 208 -rw-r--r-- 1 jenkins jenkins 0 Mar 29 2019 legacyIds -rw-r--r-- 1 jenkins jenkins 134 Jan 25 10:33 permalinks
Build 207 was previously successful and marked as "keep forever".
10:11 is when Jenkins restarted after installing plugin updates today.
This is pretty serious, as it causes our released builds to "vanish" and the deployment system to automatically downgrade to the "new" lastSuccessfulBuild.
I have not yet noticed a pattern in which jobs are affected, but so far it is only a couple of jobs that are part of a Bitbucket multi-project/branch configuration.
I just updated from 2.426.2 to 2.426.3 and it has not fixed it.
System is Rocky 9, using openjdk 17.0.10.
markewaite is there anything in the build directories you'd like to see to help work out what's wrong?
I can find a full list of plugins too. I think the problem started at some point last week, as Monday was when I noticed all the builds that had been hanging over the weekend.
jameshowe the only suggestions I have to offer are the suggestions that were offered previously. Check the system clock of the controller. Check the system clock of the computer hosting the file system. Check the system clock of the agents. Check the system clock of the Bitbucket server. Check the system clock of network devices that interact with that controller.
Check the environment variables in case something set a strange TZ value or some other value related to timekeeping.
You're welcome to upload the list of installed plugins. I think that is always a good thing to show others that you're willing to provide information that can help them help you. I don't intend to spend any time attempting to duplicate this problem. I'm confident it is a local configuration issue in your environment and only you will be able to identify the local configuration issue.
markewaite As I already said, there is no issue with the clock. There has been no timezone or any other configuration change. The problem is independent of any external system. All the timestamps on disk are correct.
Apparently at random, after a restart, Jenkins becomes unable to read the time of some builds created in the last ~2 weeks. Something is returning a `0` instead of the actual timestamp.
Where does Jenkins get the build time from? Is it from file creation/modification times, or is it stored in a file somewhere that perhaps is being corrupted, or has had an unexpected compatibility change somewhere? At least that would give me an opportunity to try to fix the existing builds.
That this has happened independently to two separate people at the same time surely suggests it is a recent code change somewhere in Jenkins? Unfortunately I cannot reliably reproduce yet, or I would start downgrading everything to find the breaking release. Edit: I misread the creation date on this ticket, sorry.
Environment:
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
INVOCATION_ID=f4fa5f8723bf4799a0184f19d1f97779
JOURNAL_STREAM=8:1058147761
NOTIFY_SOCKET=/run/systemd/notify
LOGNAME=jenkins
SYSTEMD_EXEC_PID=1151904
USER=jenkins
PWD=/var/lib/jenkins
JENKINS_HOME=/var/lib/jenkins
LANG=en_GB.UTF-8
SHLVL=0
HOME=/var/lib/jenkins
Plugins:
Additional Identities Plugin (additional-identities-plugin): 1.1
Eclipse Temurin installer Plugin (adoptopenjdk): 1.5
Analysis Model API Plugin (analysis-model-api): 11.15.0
AnsiColor (ansicolor): 1.0.4
Ant Plugin (ant): 497.v94e7d9fffa_b_9
OWASP Markup Formatter Plugin (antisamy-markup-formatter): 162.v0e6ec0fcfcf6
Apache HttpComponents Client 4.x API Plugin (apache-httpcomponents-client-4-api): 4.5.14-208.v438351942757
Artifact Deployer Plugin (artifactdeployer): 1.3
Authentication Tokens API Plugin (authentication-tokens): 1.53.v1c90fd9191a_b_
Basic Branch Build Strategies Plugin (basic-branch-build-strategies): 81.v05e333931c7d
Bitbucket Build Status Notifier Plugin (bitbucket-build-status-notifier): 1.4.2
Bitbucket Filter Project Trait (bitbucket-filter-project-trait): 1.0
Bitbucket Commit Skip SCM Behaviour (bitbucket-scm-trait-commit-skip): 0.4.0
Bitbucket Plugin (bitbucket): 241.v6d24a_57f9359
Bootstrap 5 API Plugin (bootstrap5-api): 5.3.2-3
bouncycastle API Plugin (bouncycastle-api): 2.30.1.77-225.v26ea_c9455fd9
Branch API Plugin (branch-api): 2.1144.v1425d1c3d5a_7
Build Timeout (build-timeout): 1.32
Build With Parameters (build-with-parameters): 76.v9382db_f78962
Caffeine API Plugin (caffeine-api): 3.1.8-133.v17b_1ff2e0599
Checks API plugin (checks-api): 2.0.2
Bitbucket Branch Source Plugin (cloudbees-bitbucket-branch-source): 866.vdea_7dcd3008e
Folders Plugin (cloudbees-folder): 6.858.v898218f3609d
Code Coverage Plugin (code-coverage-api): 4.99.0
Command Agent Launcher Plugin (command-launcher): 107.v773860566e2e
commons-lang3 v3.x Jenkins API Plugin (commons-lang3-api): 3.13.0-62.v7d18e55f51e2
commons-text API Plugin (commons-text-api): 1.11.0-95.v22a_d30ee5d36
Compact Columns (compact-columns): 1.185.vf3851b_4d31fe
Config File Provider Plugin (config-file-provider): 968.ve1ca_eb_913f8c
Copy Artifact Plugin (copyartifact): 722.v0662a_9b_e22a_c
Coverage Plugin (coverage): 1.10.0
Credentials Binding Plugin (credentials-binding): 657.v2b_19db_7d6e6d
Credentials Plugin (credentials): 1311.vcf0a_900b_37c2
DataTables.net API Plugin (data-tables-api): 1.13.8-2
OWASP Dependency-Check Plugin (dependency-check-jenkins-plugin): 5.4.3
Display URL API (display-url-api): 2.200.vb_9327d658781
Docker Commons Plugin (docker-commons): 439.va_3cb_0a_6a_fb_29
Docker Compose Build Step Plugin (docker-compose-build-step): 1.0
Docker Pipeline (docker-workflow): 572.v950f58993843
Durable Task Plugin (durable-task): 543.v262f6a_803410
ECharts API Plugin (echarts-api): 5.4.3-2
Email Extension Plugin (email-ext): 2.104
Extended Read Permission Plugin (extended-read-permission): 53.v6499940139e5
External Monitor Job Type Plugin (external-monitor-job): 215.v2e88e894db_f8
Extra Columns Plugin (extra-columns): 1.26
Font Awesome API Plugin (font-awesome-api): 6.5.1-2
Forensics API Plugin (forensics-api): 2.3.0
Git client plugin (git-client): 4.6.0
Git Forensics Plugin (git-forensics): 2.0.0
Git server Plugin (git-server): 114.v068a_c7cc2574
Git plugin (git): 5.2.1
GitHub API Plugin (github-api): 1.318-461.v7a_c09c9fa_d63
GitHub Branch Source Plugin (github-branch-source): 1772.va_69eda_d018d4
GitHub plugin (github): 1.37.3.1
Gradle Plugin (gradle): 2.9
Gson API Plugin (gson-api): 2.10.1-15.v0d99f670e0a_7
H2 API Plugin (h2-api): 11.1.4.199-12.v9f4244395f7a_
Handy Uri Templates 2.x API Plugin (handy-uri-templates-2-api): 2.1.8-30.v7e777411b_148
HTML Publisher plugin (htmlpublisher): 1.32
Instance Identity (instance-identity): 185.v303dc7c645f9
Ionicons API (ionicons-api): 56.v1b_1c8c49374e
Jackson 2 API Plugin (jackson2-api): 2.16.1-373.ve709c6871598
Jakarta Activation API (jakarta-activation-api): 2.0.1-3
Jakarta Mail API (jakarta-mail-api): 2.0.1-3
Javadoc Plugin (javadoc): 243.vb_b_503b_b_45537
JavaBeans Activation Framework (JAF) API (javax-activation-api): 1.2.0-6
JavaMail API (javax-mail-api): 1.6.2-9
JAXB plugin (jaxb): 2.3.9-1
Oracle Java SE Development Kit Installer Plugin (jdk-tool): 73.vddf737284550
Java JSON Web Token (JJWT) Plugin (jjwt-api): 0.11.5-77.v646c772fddb_0
jnr-posix API Plugin (jnr-posix-api): 3.1.18-1
Job Configuration History Plugin (jobConfigHistory): 1229.v3039470161a_d
Joda Time API Plugin (joda-time-api): 2.12.6-21.vca_fd74418fb_7
JQuery3 API Plugin (jquery3-api): 3.7.1-1
JSch dependency plugin (jsch): 0.2.16-86.v42e010d9484b_
JSON Path API Plugin (json-path-api): 2.9.0-33.v2527142f2e1d
JUnit Plugin (junit): 1256.v002534a_5f33e
LDAP Plugin (ldap): 711.vb_d1a_491714dc
Lockable Resources plugin (lockable-resources): 1232.v512d6c434eb_d
Log Parser Plugin (log-parser): 2.3.3
Mailer Plugin (mailer): 463.vedf8358e006b_
MapDB API Plugin (mapdb-api): 1.0.9-28.vf251ce40855d
Matrix Authorization Strategy Plugin (matrix-auth): 3.2.1
Matrix Project Plugin (matrix-project): 822.824.v14451b_c0fd42
Maven Integration plugin (maven-plugin): 3.23
Mercurial plugin (mercurial): 1260.vdfb_723cdcc81
Mina SSHD API :: Common (mina-sshd-api-common): 2.12.0-90.v9f7fb_9fa_3d3b_
Mina SSHD API :: Core (mina-sshd-api-core): 2.12.0-90.v9f7fb_9fa_3d3b_
Multiple SCMs plugin (multiple-scms): 0.8
NodeJS Plugin (nodejs): 1.6.1
OkHttp Plugin (okhttp-api): 4.11.0-157.v6852a_a_fa_ec11
PAM Authentication plugin (pam-auth): 1.10
Pipeline: Build Step (pipeline-build-step): 540.vb_e8849e1a_b_d8
Pipeline: GitHub Groovy Libraries (pipeline-github-lib): 42.v0739460cda_c4
Pipeline Graph Analysis Plugin (pipeline-graph-analysis): 202.va_d268e64deb_3
Pipeline: Groovy Libraries (pipeline-groovy-lib): 704.vc58b_8890a_384
Pipeline: Input Step (pipeline-input-step): 477.v339683a_8d55e
Pipeline Maven Plugin API (pipeline-maven-api): 1376.v18876d10ce9c
Pipeline Maven Integration Plugin (pipeline-maven): 1376.v18876d10ce9c
Pipeline: Milestone Step (pipeline-milestone-step): 111.v449306f708b_7
Pipeline: Model API (pipeline-model-api): 2.2175.v76a_fff0a_2618
Pipeline: Declarative (pipeline-model-definition): 2.2175.v76a_fff0a_2618
Pipeline: Declarative Extension Points API (pipeline-model-extensions): 2.2175.v76a_fff0a_2618
Pipeline: REST API Plugin (pipeline-rest-api): 2.34
Pipeline: Stage Step (pipeline-stage-step): 305.ve96d0205c1c6
Pipeline: Stage Tags Metadata (pipeline-stage-tags-metadata): 2.2175.v76a_fff0a_2618
Pipeline: Stage View Plugin (pipeline-stage-view): 2.34
Pipeline Utility Steps (pipeline-utility-steps): 2.16.1
Plain Credentials Plugin (plain-credentials): 143.v1b_df8b_d3b_e48
Plugin Utilities API Plugin (plugin-util-api): 3.8.0
Prism API Plugin (prism-api): 1.29.0-10
Publish Over SSH (publish-over-ssh): 1.25
Infrastructure plugin for Publish Over X (publish-over): 0.22
Pyenv Pipeline Plugin (pyenv-pipeline): 2.1.2
Resource Disposer Plugin (resource-disposer): 0.23
SCM API Plugin (scm-api): 683.vb_16722fb_b_80b_
Script Security Plugin (script-security): 1313.v7a_6067dc7087
ShiningPanda Plugin (shiningpanda): 0.24
Slack Notification Plugin (slack): 684.v833089650554
SnakeYAML API Plugin (snakeyaml-api): 2.2-111.vc6598e30cc65
SSH Credentials Plugin (ssh-credentials): 308.ve4497b_ccd8f4
SSH Build Agents plugin (ssh-slaves): 2.948.vb_8050d697fec
SSH server (sshd): 3.322.v159e91f6a_550
Structs Plugin (structs): 337.v1b_04ea_4df7c8
Subversion Plug-in (subversion): 2.17.3
TAP Plugin (tap): 2.4.1
Timestamper (timestamper): 1.26
Token Macro Plugin (token-macro): 400.v35420b_922dcb_
Trilead API Plugin (trilead-api): 2.133.vfb_8a_7b_9c5dd1
Variant Plugin (variant): 60.v7290fc0eb_b_cd
Versions Node Monitors plugin (versioncolumn): 233.v2d198f8212a_2
Warnings Plugin (warnings-ng): 10.7.0
Pipeline (workflow-aggregator): 596.v8c21c963d92d
Pipeline: API (workflow-api): 1283.v99c10937efcb_
Pipeline: Basic Steps (workflow-basic-steps): 1042.ve7b_140c4a_e0c
Pipeline: Groovy (workflow-cps): 3853.vb_a_490d892963
Pipeline: Nodes and Processes (workflow-durable-task-step): 1317.v5337e0c1fe28
Pipeline: Job (workflow-job): 1385.vb_58b_86ea_fff1
Pipeline: Multibranch (workflow-multibranch): 773.vc4fe1378f1d5
Pipeline: SCM Step (workflow-scm-step): 415.v434365564324
Pipeline: Step API (workflow-step-api): 657.v03b_e8115821b_
Pipeline: Supporting APIs (workflow-support): 865.v43e78cc44e0d
Workspace Cleanup Plugin (ws-cleanup): 0.45
Though I have just noticed it's saying BST in the UI, despite the dates being in the middle of winter...
The server is set to Europe/London, and is showing the correct GMT times in the listing above. My Jenkins user preference is also set to Europe/London.
The build.xml for 207 shown above is:
<?xml version='1.1' encoding='UTF-8'?>
<flow-build plugin="workflow-job@1385.vb_58b_86ea_fff1">
<actions/>
<queueId>-1</queueId>
<timestamp>0</timestamp>
<startTime>0</startTime>
<result>FAILURE</result>
<duration>0</duration>
<keepLog>false</keepLog>
<completed>true</completed>
</flow-build>
This job definitely succeeded on the 17th. The log showing that is right there in the directory.
A restart of Jenkins on the 23rd has rewritten this file.
We had to scrap the instance and build a new one then. So far, its not occurred again. But we were not able to resolve, checked the system clock as well. We used Amazon linux on EKS managed nodes, so I doubt its system clock.
It seems most likely that there is a system configuration error that is causing the problem. For example, the system clock may be incorrect or may have been dramatically adjusted. I don't think there is anything Jenkins can do about this other than advise that Jenkins controller and Jenkins agent clocks should be correct and consistent with one another. Programs like ntpd and chrony help Unix users keep time correct. Windows users can configure the operating system to use a time server as well.