Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-66328

pipeline in Jenkins are showing like executed back to 1 Jan 1970

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • core
    • None
    • Jenkins 2.289.3

      Hi, 

      Jenkins is deployed on EKS.

      Everytime Jenkins is redployed, the past jobs are showing dated "1 Jan 1970".

      All past jobs are in hanged mode, each time needing to abort from script console. 

          [JENKINS-66328] pipeline in Jenkins are showing like executed back to 1 Jan 1970

          Pinned comments

          Pinned by James Howe

          James Howe added a comment - - edited

          The build.xml for 207 shown above is:

          <?xml version='1.1' encoding='UTF-8'?>
          <flow-build plugin="workflow-job@1385.vb_58b_86ea_fff1">
            <actions/>
            <queueId>-1</queueId>
            <timestamp>0</timestamp>
            <startTime>0</startTime>
            <result>FAILURE</result>
            <duration>0</duration>
            <keepLog>false</keepLog>
            <completed>true</completed>
          </flow-build>

          This job definitely succeeded on the 17th. The log showing that is right there in the directory.
          A restart of Jenkins on the 23rd has rewritten this file.

          James Howe added a comment - - edited The build.xml for 207 shown above is: <?xml version='1.1' encoding='UTF-8'?> <flow-build plugin="workflow-job@1385.vb_58b_86ea_fff1">   <actions/>   <queueId>-1</queueId>   <timestamp>0</timestamp>   <startTime>0</startTime>   <result>FAILURE</result>   <duration>0</duration>   <keepLog>false</keepLog>   <completed>true</completed> </flow-build> This job definitely succeeded on the 17th. The log showing that is right there in the directory. A restart of Jenkins on the 23rd has rewritten this file.

          Pinned by James Howe

          Mark Waite added a comment -

          Can each of those that are affected by the problem please provide the detailed list of installed plugins and their versions?

          https://github.com/jenkins-infra/helpdesk/issues/3934#issuecomment-1932685147 indicates that the datadog plugin 6.0.0 may be the cause of some of the issues, though it is not yet clear if it is the cause of all of the issues.

          Mark Waite added a comment - Can each of those that are affected by the problem please provide the detailed list of installed plugins and their versions? https://github.com/jenkins-infra/helpdesk/issues/3934#issuecomment-1932685147 indicates that the datadog plugin 6.0.0 may be the cause of some of the issues, though it is not yet clear if it is the cause of all of the issues.

          All comments

          Mark Waite added a comment -

          It seems most likely that there is a system configuration error that is causing the problem. For example, the system clock may be incorrect or may have been dramatically adjusted. I don't think there is anything Jenkins can do about this other than advise that Jenkins controller and Jenkins agent clocks should be correct and consistent with one another. Programs like ntpd and chrony help Unix users keep time correct. Windows users can configure the operating system to use a time server as well.

          Mark Waite added a comment - It seems most likely that there is a system configuration error that is causing the problem. For example, the system clock may be incorrect or may have been dramatically adjusted. I don't think there is anything Jenkins can do about this other than advise that Jenkins controller and Jenkins agent clocks should be correct and consistent with one another. Programs like ntpd and chrony help Unix users keep time correct. Windows users can configure the operating system to use a time server as well.

          James Howe added a comment - - edited

          I am having the same problem. The builds all look fine on disk, but Jenkins considers them failed and hangs on running a new build until a second restart.

          There is no clock problem detectable on the machine.

          drwxr-xr-x 7 jenkins jenkins 4096 Dec 13 17:37 204
          drwxr-xr-x 4 jenkins jenkins 4096 Jan  5 17:25 205
          drwxr-xr-x 6 jenkins jenkins 4096 Jan  5 17:50 206
          drwxr-xr-x 6 jenkins jenkins 4096 Jan 23 11:07 207
          drwxr-xr-x 6 jenkins jenkins 4096 Jan 25 10:11 208
          -rw-r--r-- 1 jenkins jenkins    0 Mar 29  2019 legacyIds
          -rw-r--r-- 1 jenkins jenkins  134 Jan 25 10:33 permalinks
          

          Build 207 was previously successful and marked as "keep forever".
          10:11 is when Jenkins restarted after installing plugin updates today.

          This is pretty serious, as it causes our released builds to "vanish" and the deployment system to automatically downgrade to the "new" lastSuccessfulBuild.

          I have not yet noticed a pattern in which jobs are affected, but so far it is only a couple of jobs that are part of a Bitbucket multi-project/branch configuration.

          I just updated from 2.426.2 to 2.426.3 and it has not fixed it.
          System is Rocky 9, using openjdk 17.0.10.

          James Howe added a comment - - edited I am having the same problem. The builds all look fine on disk, but Jenkins considers them failed and hangs on running a new build until a second restart. There is no clock problem detectable on the machine. drwxr-xr-x 7 jenkins jenkins 4096 Dec 13 17:37 204 drwxr-xr-x 4 jenkins jenkins 4096 Jan 5 17:25 205 drwxr-xr-x 6 jenkins jenkins 4096 Jan 5 17:50 206 drwxr-xr-x 6 jenkins jenkins 4096 Jan 23 11:07 207 drwxr-xr-x 6 jenkins jenkins 4096 Jan 25 10:11 208 -rw-r--r-- 1 jenkins jenkins    0 Mar 29  2019 legacyIds -rw-r--r-- 1 jenkins jenkins 134 Jan 25 10:33 permalinks Build 207 was previously successful and marked as "keep forever". 10:11 is when Jenkins restarted after installing plugin updates today. This is pretty serious, as it causes our released builds to "vanish" and the deployment system to automatically downgrade to the "new" lastSuccessfulBuild. I have not yet noticed a pattern in which jobs are affected, but so far it is only a couple of jobs that are part of a Bitbucket multi-project/branch configuration. I just updated from 2.426.2 to 2.426.3 and it has not fixed it. System is Rocky 9, using openjdk 17.0.10.

          James Howe added a comment -

          markewaite is there anything in the build directories you'd like to see to help work out what's wrong?

          I can find a full list of plugins too. I think the problem started at some point last week, as Monday was when I noticed all the builds that had been hanging over the weekend.

          James Howe added a comment - markewaite is there anything in the build directories you'd like to see to help work out what's wrong? I can find a full list of plugins too. I think the problem started at some point last week, as Monday was when I noticed all the builds that had been hanging over the weekend.

          Mark Waite added a comment -

          jameshowe the only suggestions I have to offer are the suggestions that were offered previously. Check the system clock of the controller. Check the system clock of the computer hosting the file system. Check the system clock of the agents. Check the system clock of the Bitbucket server. Check the system clock of network devices that interact with that controller.

          Check the environment variables in case something set a strange TZ value or some other value related to timekeeping.

          You're welcome to upload the list of installed plugins. I think that is always a good thing to show others that you're willing to provide information that can help them help you. I don't intend to spend any time attempting to duplicate this problem. I'm confident it is a local configuration issue in your environment and only you will be able to identify the local configuration issue.

          Mark Waite added a comment - jameshowe the only suggestions I have to offer are the suggestions that were offered previously. Check the system clock of the controller. Check the system clock of the computer hosting the file system. Check the system clock of the agents. Check the system clock of the Bitbucket server. Check the system clock of network devices that interact with that controller. Check the environment variables in case something set a strange TZ value or some other value related to timekeeping. You're welcome to upload the list of installed plugins. I think that is always a good thing to show others that you're willing to provide information that can help them help you. I don't intend to spend any time attempting to duplicate this problem. I'm confident it is a local configuration issue in your environment and only you will be able to identify the local configuration issue.

          James Howe added a comment - - edited

          markewaite As I already said, there is no issue with the clock. There has been no timezone or any other configuration change. The problem is independent of any external system. All the timestamps on disk are correct.

          Apparently at random, after a restart, Jenkins becomes unable to read the time of some builds created in the last ~2 weeks. Something is returning a `0` instead of the actual timestamp.

          Where does Jenkins get the build time from? Is it from file creation/modification times, or is it stored in a file somewhere that perhaps is being corrupted, or has had an unexpected compatibility change somewhere? At least that would give me an opportunity to try to fix the existing builds.

          That this has happened independently to two separate people at the same time surely suggests it is a recent code change somewhere in Jenkins? Unfortunately I cannot reliably reproduce yet, or I would start downgrading everything to find the breaking release. Edit: I misread the creation date on this ticket, sorry.


          Environment:

          PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
          INVOCATION_ID=f4fa5f8723bf4799a0184f19d1f97779
          JOURNAL_STREAM=8:1058147761
          NOTIFY_SOCKET=/run/systemd/notify
          LOGNAME=jenkins
          SYSTEMD_EXEC_PID=1151904
          USER=jenkins
          PWD=/var/lib/jenkins
          JENKINS_HOME=/var/lib/jenkins
          LANG=en_GB.UTF-8
          SHLVL=0
          HOME=/var/lib/jenkins

          Plugins:

          Additional Identities Plugin (additional-identities-plugin): 1.1
          Eclipse Temurin installer Plugin (adoptopenjdk): 1.5
          Analysis Model API Plugin (analysis-model-api): 11.15.0
          AnsiColor (ansicolor): 1.0.4
          Ant Plugin (ant): 497.v94e7d9fffa_b_9
          OWASP Markup Formatter Plugin (antisamy-markup-formatter): 162.v0e6ec0fcfcf6
          Apache HttpComponents Client 4.x API Plugin (apache-httpcomponents-client-4-api): 4.5.14-208.v438351942757
          Artifact Deployer Plugin (artifactdeployer): 1.3
          Authentication Tokens API Plugin (authentication-tokens): 1.53.v1c90fd9191a_b_
          Basic Branch Build Strategies Plugin (basic-branch-build-strategies): 81.v05e333931c7d
          Bitbucket Build Status Notifier Plugin (bitbucket-build-status-notifier): 1.4.2
          Bitbucket Filter Project Trait (bitbucket-filter-project-trait): 1.0
          Bitbucket Commit Skip SCM Behaviour (bitbucket-scm-trait-commit-skip): 0.4.0
          Bitbucket Plugin (bitbucket): 241.v6d24a_57f9359
          Bootstrap 5 API Plugin (bootstrap5-api): 5.3.2-3
          bouncycastle API Plugin (bouncycastle-api): 2.30.1.77-225.v26ea_c9455fd9
          Branch API Plugin (branch-api): 2.1144.v1425d1c3d5a_7
          Build Timeout (build-timeout): 1.32
          Build With Parameters (build-with-parameters): 76.v9382db_f78962
          Caffeine API Plugin (caffeine-api): 3.1.8-133.v17b_1ff2e0599
          Checks API plugin (checks-api): 2.0.2
          Bitbucket Branch Source Plugin (cloudbees-bitbucket-branch-source): 866.vdea_7dcd3008e
          Folders Plugin (cloudbees-folder): 6.858.v898218f3609d
          Code Coverage Plugin (code-coverage-api): 4.99.0
          Command Agent Launcher Plugin (command-launcher): 107.v773860566e2e
          commons-lang3 v3.x Jenkins API Plugin (commons-lang3-api): 3.13.0-62.v7d18e55f51e2
          commons-text API Plugin (commons-text-api): 1.11.0-95.v22a_d30ee5d36
          Compact Columns (compact-columns): 1.185.vf3851b_4d31fe
          Config File Provider Plugin (config-file-provider): 968.ve1ca_eb_913f8c
          Copy Artifact Plugin (copyartifact): 722.v0662a_9b_e22a_c
          Coverage Plugin (coverage): 1.10.0
          Credentials Binding Plugin (credentials-binding): 657.v2b_19db_7d6e6d
          Credentials Plugin (credentials): 1311.vcf0a_900b_37c2
          DataTables.net API Plugin (data-tables-api): 1.13.8-2
          OWASP Dependency-Check Plugin (dependency-check-jenkins-plugin): 5.4.3
          Display URL API (display-url-api): 2.200.vb_9327d658781
          Docker Commons Plugin (docker-commons): 439.va_3cb_0a_6a_fb_29
          Docker Compose Build Step Plugin (docker-compose-build-step): 1.0
          Docker Pipeline (docker-workflow): 572.v950f58993843
          Durable Task Plugin (durable-task): 543.v262f6a_803410
          ECharts API Plugin (echarts-api): 5.4.3-2
          Email Extension Plugin (email-ext): 2.104
          Extended Read Permission Plugin (extended-read-permission): 53.v6499940139e5
          External Monitor Job Type Plugin (external-monitor-job): 215.v2e88e894db_f8
          Extra Columns Plugin (extra-columns): 1.26
          Font Awesome API Plugin (font-awesome-api): 6.5.1-2
          Forensics API Plugin (forensics-api): 2.3.0
          Git client plugin (git-client): 4.6.0
          Git Forensics Plugin (git-forensics): 2.0.0
          Git server Plugin (git-server): 114.v068a_c7cc2574
          Git plugin (git): 5.2.1
          GitHub API Plugin (github-api): 1.318-461.v7a_c09c9fa_d63
          GitHub Branch Source Plugin (github-branch-source): 1772.va_69eda_d018d4
          GitHub plugin (github): 1.37.3.1
          Gradle Plugin (gradle): 2.9
          Gson API Plugin (gson-api): 2.10.1-15.v0d99f670e0a_7
          H2 API Plugin (h2-api): 11.1.4.199-12.v9f4244395f7a_
          Handy Uri Templates 2.x API Plugin (handy-uri-templates-2-api): 2.1.8-30.v7e777411b_148
          HTML Publisher plugin (htmlpublisher): 1.32
          Instance Identity (instance-identity): 185.v303dc7c645f9
          Ionicons API (ionicons-api): 56.v1b_1c8c49374e
          Jackson 2 API Plugin (jackson2-api): 2.16.1-373.ve709c6871598
          Jakarta Activation API (jakarta-activation-api): 2.0.1-3
          Jakarta Mail API (jakarta-mail-api): 2.0.1-3
          Javadoc Plugin (javadoc): 243.vb_b_503b_b_45537
          JavaBeans Activation Framework (JAF) API (javax-activation-api): 1.2.0-6
          JavaMail API (javax-mail-api): 1.6.2-9
          JAXB plugin (jaxb): 2.3.9-1
          Oracle Java SE Development Kit Installer Plugin (jdk-tool): 73.vddf737284550
          Java JSON Web Token (JJWT) Plugin (jjwt-api): 0.11.5-77.v646c772fddb_0
          jnr-posix API Plugin (jnr-posix-api): 3.1.18-1
          Job Configuration History Plugin (jobConfigHistory): 1229.v3039470161a_d
          Joda Time API Plugin (joda-time-api): 2.12.6-21.vca_fd74418fb_7
          JQuery3 API Plugin (jquery3-api): 3.7.1-1
          JSch dependency plugin (jsch): 0.2.16-86.v42e010d9484b_
          JSON Path API Plugin (json-path-api): 2.9.0-33.v2527142f2e1d
          JUnit Plugin (junit): 1256.v002534a_5f33e
          LDAP Plugin (ldap): 711.vb_d1a_491714dc
          Lockable Resources plugin (lockable-resources): 1232.v512d6c434eb_d
          Log Parser Plugin (log-parser): 2.3.3
          Mailer Plugin (mailer): 463.vedf8358e006b_
          MapDB API Plugin (mapdb-api): 1.0.9-28.vf251ce40855d
          Matrix Authorization Strategy Plugin (matrix-auth): 3.2.1
          Matrix Project Plugin (matrix-project): 822.824.v14451b_c0fd42
          Maven Integration plugin (maven-plugin): 3.23
          Mercurial plugin (mercurial): 1260.vdfb_723cdcc81
          Mina SSHD API :: Common (mina-sshd-api-common): 2.12.0-90.v9f7fb_9fa_3d3b_
          Mina SSHD API :: Core (mina-sshd-api-core): 2.12.0-90.v9f7fb_9fa_3d3b_
          Multiple SCMs plugin (multiple-scms): 0.8
          NodeJS Plugin (nodejs): 1.6.1
          OkHttp Plugin (okhttp-api): 4.11.0-157.v6852a_a_fa_ec11
          PAM Authentication plugin (pam-auth): 1.10
          Pipeline: Build Step (pipeline-build-step): 540.vb_e8849e1a_b_d8
          Pipeline: GitHub Groovy Libraries (pipeline-github-lib): 42.v0739460cda_c4
          Pipeline Graph Analysis Plugin (pipeline-graph-analysis): 202.va_d268e64deb_3
          Pipeline: Groovy Libraries (pipeline-groovy-lib): 704.vc58b_8890a_384
          Pipeline: Input Step (pipeline-input-step): 477.v339683a_8d55e
          Pipeline Maven Plugin API (pipeline-maven-api): 1376.v18876d10ce9c
          Pipeline Maven Integration Plugin (pipeline-maven): 1376.v18876d10ce9c
          Pipeline: Milestone Step (pipeline-milestone-step): 111.v449306f708b_7
          Pipeline: Model API (pipeline-model-api): 2.2175.v76a_fff0a_2618
          Pipeline: Declarative (pipeline-model-definition): 2.2175.v76a_fff0a_2618
          Pipeline: Declarative Extension Points API (pipeline-model-extensions): 2.2175.v76a_fff0a_2618
          Pipeline: REST API Plugin (pipeline-rest-api): 2.34
          Pipeline: Stage Step (pipeline-stage-step): 305.ve96d0205c1c6
          Pipeline: Stage Tags Metadata (pipeline-stage-tags-metadata): 2.2175.v76a_fff0a_2618
          Pipeline: Stage View Plugin (pipeline-stage-view): 2.34
          Pipeline Utility Steps (pipeline-utility-steps): 2.16.1
          Plain Credentials Plugin (plain-credentials): 143.v1b_df8b_d3b_e48
          Plugin Utilities API Plugin (plugin-util-api): 3.8.0
          Prism API Plugin (prism-api): 1.29.0-10
          Publish Over SSH (publish-over-ssh): 1.25
          Infrastructure plugin for Publish Over X (publish-over): 0.22
          Pyenv Pipeline Plugin (pyenv-pipeline): 2.1.2
          Resource Disposer Plugin (resource-disposer): 0.23
          SCM API Plugin (scm-api): 683.vb_16722fb_b_80b_
          Script Security Plugin (script-security): 1313.v7a_6067dc7087
          ShiningPanda Plugin (shiningpanda): 0.24
          Slack Notification Plugin (slack): 684.v833089650554
          SnakeYAML API Plugin (snakeyaml-api): 2.2-111.vc6598e30cc65
          SSH Credentials Plugin (ssh-credentials): 308.ve4497b_ccd8f4
          SSH Build Agents plugin (ssh-slaves): 2.948.vb_8050d697fec
          SSH server (sshd): 3.322.v159e91f6a_550
          Structs Plugin (structs): 337.v1b_04ea_4df7c8
          Subversion Plug-in (subversion): 2.17.3
          TAP Plugin (tap): 2.4.1
          Timestamper (timestamper): 1.26
          Token Macro Plugin (token-macro): 400.v35420b_922dcb_
          Trilead API Plugin (trilead-api): 2.133.vfb_8a_7b_9c5dd1
          Variant Plugin (variant): 60.v7290fc0eb_b_cd
          Versions Node Monitors plugin (versioncolumn): 233.v2d198f8212a_2
          Warnings Plugin (warnings-ng): 10.7.0
          Pipeline (workflow-aggregator): 596.v8c21c963d92d
          Pipeline: API (workflow-api): 1283.v99c10937efcb_
          Pipeline: Basic Steps (workflow-basic-steps): 1042.ve7b_140c4a_e0c
          Pipeline: Groovy (workflow-cps): 3853.vb_a_490d892963
          Pipeline: Nodes and Processes (workflow-durable-task-step): 1317.v5337e0c1fe28
          Pipeline: Job (workflow-job): 1385.vb_58b_86ea_fff1
          Pipeline: Multibranch (workflow-multibranch): 773.vc4fe1378f1d5
          Pipeline: SCM Step (workflow-scm-step): 415.v434365564324
          Pipeline: Step API (workflow-step-api): 657.v03b_e8115821b_
          Pipeline: Supporting APIs (workflow-support): 865.v43e78cc44e0d
          Workspace Cleanup Plugin (ws-cleanup): 0.45

          James Howe added a comment - - edited markewaite As I already said, there is no issue with the clock. There has been no timezone or any other configuration change. The problem is independent of any external system. All the timestamps on disk are correct. Apparently at random, after a restart, Jenkins becomes unable to read the time of some builds created in the last ~2 weeks. Something is returning a `0` instead of the actual timestamp. Where does Jenkins get the build time from? Is it from file creation/modification times, or is it stored in a file somewhere that perhaps is being corrupted, or has had an unexpected compatibility change somewhere? At least that would give me an opportunity to try to fix the existing builds. That this has happened independently to two separate people at the same time surely suggests it is a recent code change somewhere in Jenkins? Unfortunately I cannot reliably reproduce yet, or I would start downgrading everything to find the breaking release. Edit: I misread the creation date on this ticket, sorry. Environment: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin INVOCATION_ID=f4fa5f8723bf4799a0184f19d1f97779 JOURNAL_STREAM=8:1058147761 NOTIFY_SOCKET=/run/systemd/notify LOGNAME=jenkins SYSTEMD_EXEC_PID=1151904 USER=jenkins PWD=/var/lib/jenkins JENKINS_HOME=/var/lib/jenkins LANG=en_GB.UTF-8 SHLVL=0 HOME=/var/lib/jenkins Plugins: Additional Identities Plugin (additional-identities-plugin): 1.1 Eclipse Temurin installer Plugin (adoptopenjdk): 1.5 Analysis Model API Plugin (analysis-model-api): 11.15.0 AnsiColor (ansicolor): 1.0.4 Ant Plugin (ant): 497.v94e7d9fffa_b_9 OWASP Markup Formatter Plugin (antisamy-markup-formatter): 162.v0e6ec0fcfcf6 Apache HttpComponents Client 4.x API Plugin (apache-httpcomponents-client-4-api): 4.5.14-208.v438351942757 Artifact Deployer Plugin (artifactdeployer): 1.3 Authentication Tokens API Plugin (authentication-tokens): 1.53.v1c90fd9191a_b_ Basic Branch Build Strategies Plugin (basic-branch-build-strategies): 81.v05e333931c7d Bitbucket Build Status Notifier Plugin (bitbucket-build-status-notifier): 1.4.2 Bitbucket Filter Project Trait (bitbucket-filter-project-trait): 1.0 Bitbucket Commit Skip SCM Behaviour (bitbucket-scm-trait-commit-skip): 0.4.0 Bitbucket Plugin (bitbucket): 241.v6d24a_57f9359 Bootstrap 5 API Plugin (bootstrap5-api): 5.3.2-3 bouncycastle API Plugin (bouncycastle-api): 2.30.1.77-225.v26ea_c9455fd9 Branch API Plugin (branch-api): 2.1144.v1425d1c3d5a_7 Build Timeout (build-timeout): 1.32 Build With Parameters (build-with-parameters): 76.v9382db_f78962 Caffeine API Plugin (caffeine-api): 3.1.8-133.v17b_1ff2e0599 Checks API plugin (checks-api): 2.0.2 Bitbucket Branch Source Plugin (cloudbees-bitbucket-branch-source): 866.vdea_7dcd3008e Folders Plugin (cloudbees-folder): 6.858.v898218f3609d Code Coverage Plugin (code-coverage-api): 4.99.0 Command Agent Launcher Plugin (command-launcher): 107.v773860566e2e commons-lang3 v3.x Jenkins API Plugin (commons-lang3-api): 3.13.0-62.v7d18e55f51e2 commons-text API Plugin (commons-text-api): 1.11.0-95.v22a_d30ee5d36 Compact Columns (compact-columns): 1.185.vf3851b_4d31fe Config File Provider Plugin (config-file-provider): 968.ve1ca_eb_913f8c Copy Artifact Plugin (copyartifact): 722.v0662a_9b_e22a_c Coverage Plugin (coverage): 1.10.0 Credentials Binding Plugin (credentials-binding): 657.v2b_19db_7d6e6d Credentials Plugin (credentials): 1311.vcf0a_900b_37c2 DataTables.net API Plugin (data-tables-api): 1.13.8-2 OWASP Dependency-Check Plugin (dependency-check-jenkins-plugin): 5.4.3 Display URL API (display-url-api): 2.200.vb_9327d658781 Docker Commons Plugin (docker-commons): 439.va_3cb_0a_6a_fb_29 Docker Compose Build Step Plugin (docker-compose-build-step): 1.0 Docker Pipeline (docker-workflow): 572.v950f58993843 Durable Task Plugin (durable-task): 543.v262f6a_803410 ECharts API Plugin (echarts-api): 5.4.3-2 Email Extension Plugin (email-ext): 2.104 Extended Read Permission Plugin (extended-read-permission): 53.v6499940139e5 External Monitor Job Type Plugin (external-monitor-job): 215.v2e88e894db_f8 Extra Columns Plugin (extra-columns): 1.26 Font Awesome API Plugin (font-awesome-api): 6.5.1-2 Forensics API Plugin (forensics-api): 2.3.0 Git client plugin (git-client): 4.6.0 Git Forensics Plugin (git-forensics): 2.0.0 Git server Plugin (git-server): 114.v068a_c7cc2574 Git plugin (git): 5.2.1 GitHub API Plugin (github-api): 1.318-461.v7a_c09c9fa_d63 GitHub Branch Source Plugin (github-branch-source): 1772.va_69eda_d018d4 GitHub plugin (github): 1.37.3.1 Gradle Plugin (gradle): 2.9 Gson API Plugin (gson-api): 2.10.1-15.v0d99f670e0a_7 H2 API Plugin (h2-api): 11.1.4.199-12.v9f4244395f7a_ Handy Uri Templates 2.x API Plugin (handy-uri-templates-2-api): 2.1.8-30.v7e777411b_148 HTML Publisher plugin (htmlpublisher): 1.32 Instance Identity (instance-identity): 185.v303dc7c645f9 Ionicons API (ionicons-api): 56.v1b_1c8c49374e Jackson 2 API Plugin (jackson2-api): 2.16.1-373.ve709c6871598 Jakarta Activation API (jakarta-activation-api): 2.0.1-3 Jakarta Mail API (jakarta-mail-api): 2.0.1-3 Javadoc Plugin (javadoc): 243.vb_b_503b_b_45537 JavaBeans Activation Framework (JAF) API (javax-activation-api): 1.2.0-6 JavaMail API (javax-mail-api): 1.6.2-9 JAXB plugin (jaxb): 2.3.9-1 Oracle Java SE Development Kit Installer Plugin (jdk-tool): 73.vddf737284550 Java JSON Web Token (JJWT) Plugin (jjwt-api): 0.11.5-77.v646c772fddb_0 jnr-posix API Plugin (jnr-posix-api): 3.1.18-1 Job Configuration History Plugin (jobConfigHistory): 1229.v3039470161a_d Joda Time API Plugin (joda-time-api): 2.12.6-21.vca_fd74418fb_7 JQuery3 API Plugin (jquery3-api): 3.7.1-1 JSch dependency plugin (jsch): 0.2.16-86.v42e010d9484b_ JSON Path API Plugin (json-path-api): 2.9.0-33.v2527142f2e1d JUnit Plugin (junit): 1256.v002534a_5f33e LDAP Plugin (ldap): 711.vb_d1a_491714dc Lockable Resources plugin (lockable-resources): 1232.v512d6c434eb_d Log Parser Plugin (log-parser): 2.3.3 Mailer Plugin (mailer): 463.vedf8358e006b_ MapDB API Plugin (mapdb-api): 1.0.9-28.vf251ce40855d Matrix Authorization Strategy Plugin (matrix-auth): 3.2.1 Matrix Project Plugin (matrix-project): 822.824.v14451b_c0fd42 Maven Integration plugin (maven-plugin): 3.23 Mercurial plugin (mercurial): 1260.vdfb_723cdcc81 Mina SSHD API :: Common (mina-sshd-api-common): 2.12.0-90.v9f7fb_9fa_3d3b_ Mina SSHD API :: Core (mina-sshd-api-core): 2.12.0-90.v9f7fb_9fa_3d3b_ Multiple SCMs plugin (multiple-scms): 0.8 NodeJS Plugin (nodejs): 1.6.1 OkHttp Plugin (okhttp-api): 4.11.0-157.v6852a_a_fa_ec11 PAM Authentication plugin (pam-auth): 1.10 Pipeline: Build Step (pipeline-build-step): 540.vb_e8849e1a_b_d8 Pipeline: GitHub Groovy Libraries (pipeline-github-lib): 42.v0739460cda_c4 Pipeline Graph Analysis Plugin (pipeline-graph-analysis): 202.va_d268e64deb_3 Pipeline: Groovy Libraries (pipeline-groovy-lib): 704.vc58b_8890a_384 Pipeline: Input Step (pipeline-input-step): 477.v339683a_8d55e Pipeline Maven Plugin API (pipeline-maven-api): 1376.v18876d10ce9c Pipeline Maven Integration Plugin (pipeline-maven): 1376.v18876d10ce9c Pipeline: Milestone Step (pipeline-milestone-step): 111.v449306f708b_7 Pipeline: Model API (pipeline-model-api): 2.2175.v76a_fff0a_2618 Pipeline: Declarative (pipeline-model-definition): 2.2175.v76a_fff0a_2618 Pipeline: Declarative Extension Points API (pipeline-model-extensions): 2.2175.v76a_fff0a_2618 Pipeline: REST API Plugin (pipeline-rest-api): 2.34 Pipeline: Stage Step (pipeline-stage-step): 305.ve96d0205c1c6 Pipeline: Stage Tags Metadata (pipeline-stage-tags-metadata): 2.2175.v76a_fff0a_2618 Pipeline: Stage View Plugin (pipeline-stage-view): 2.34 Pipeline Utility Steps (pipeline-utility-steps): 2.16.1 Plain Credentials Plugin (plain-credentials): 143.v1b_df8b_d3b_e48 Plugin Utilities API Plugin (plugin-util-api): 3.8.0 Prism API Plugin (prism-api): 1.29.0-10 Publish Over SSH (publish-over-ssh): 1.25 Infrastructure plugin for Publish Over X (publish-over): 0.22 Pyenv Pipeline Plugin (pyenv-pipeline): 2.1.2 Resource Disposer Plugin (resource-disposer): 0.23 SCM API Plugin (scm-api): 683.vb_16722fb_b_80b_ Script Security Plugin (script-security): 1313.v7a_6067dc7087 ShiningPanda Plugin (shiningpanda): 0.24 Slack Notification Plugin (slack): 684.v833089650554 SnakeYAML API Plugin (snakeyaml-api): 2.2-111.vc6598e30cc65 SSH Credentials Plugin (ssh-credentials): 308.ve4497b_ccd8f4 SSH Build Agents plugin (ssh-slaves): 2.948.vb_8050d697fec SSH server (sshd): 3.322.v159e91f6a_550 Structs Plugin (structs): 337.v1b_04ea_4df7c8 Subversion Plug-in (subversion): 2.17.3 TAP Plugin (tap): 2.4.1 Timestamper (timestamper): 1.26 Token Macro Plugin (token-macro): 400.v35420b_922dcb_ Trilead API Plugin (trilead-api): 2.133.vfb_8a_7b_9c5dd1 Variant Plugin (variant): 60.v7290fc0eb_b_cd Versions Node Monitors plugin (versioncolumn): 233.v2d198f8212a_2 Warnings Plugin (warnings-ng): 10.7.0 Pipeline (workflow-aggregator): 596.v8c21c963d92d Pipeline: API (workflow-api): 1283.v99c10937efcb_ Pipeline: Basic Steps (workflow-basic-steps): 1042.ve7b_140c4a_e0c Pipeline: Groovy (workflow-cps): 3853.vb_a_490d892963 Pipeline: Nodes and Processes (workflow-durable-task-step): 1317.v5337e0c1fe28 Pipeline: Job (workflow-job): 1385.vb_58b_86ea_fff1 Pipeline: Multibranch (workflow-multibranch): 773.vc4fe1378f1d5 Pipeline: SCM Step (workflow-scm-step): 415.v434365564324 Pipeline: Step API (workflow-step-api): 657.v03b_e8115821b_ Pipeline: Supporting APIs (workflow-support): 865.v43e78cc44e0d Workspace Cleanup Plugin (ws-cleanup): 0.45

          James Howe added a comment - - edited

          Though I have just noticed it's saying BST in the UI, despite the dates being in the middle of winter...

          The server is set to Europe/London, and is showing the correct GMT times in the listing above. My Jenkins user preference is also set to Europe/London.

          James Howe added a comment - - edited Though I have just noticed it's saying BST in the UI, despite the dates being in the middle of winter... The server is set to Europe/London, and is showing the correct GMT times in the listing above. My Jenkins user preference is also set to Europe/London.

          Pinned by James Howe

          James Howe added a comment - - edited

          The build.xml for 207 shown above is:

          <?xml version='1.1' encoding='UTF-8'?>
          <flow-build plugin="workflow-job@1385.vb_58b_86ea_fff1">
            <actions/>
            <queueId>-1</queueId>
            <timestamp>0</timestamp>
            <startTime>0</startTime>
            <result>FAILURE</result>
            <duration>0</duration>
            <keepLog>false</keepLog>
            <completed>true</completed>
          </flow-build>

          This job definitely succeeded on the 17th. The log showing that is right there in the directory.
          A restart of Jenkins on the 23rd has rewritten this file.

          James Howe added a comment - - edited The build.xml for 207 shown above is: <?xml version='1.1' encoding='UTF-8'?> <flow-build plugin="workflow-job@1385.vb_58b_86ea_fff1">   <actions/>   <queueId>-1</queueId>   <timestamp>0</timestamp>   <startTime>0</startTime>   <result>FAILURE</result>   <duration>0</duration>   <keepLog>false</keepLog>   <completed>true</completed> </flow-build> This job definitely succeeded on the 17th. The log showing that is right there in the directory. A restart of Jenkins on the 23rd has rewritten this file.

          Kishore added a comment -

          We had to scrap the instance and build a new one then. So far, its not occurred again. But we were not able to resolve, checked the system clock as well.  We used Amazon linux on EKS managed nodes, so I doubt its system clock.

          Kishore added a comment - We had to scrap the instance and build a new one then. So far, its not occurred again. But we were not able to resolve, checked the system clock as well.  We used Amazon linux on EKS managed nodes, so I doubt its system clock.

          Peter Simms added a comment -

          We are seeing the same issue here from a recent upgrade to 2.426.3 from 2.426.1

          Peter Simms added a comment - We are seeing the same issue here from a recent upgrade to 2.426.3 from 2.426.1

          James Howe added a comment - - edited

          I had it start while on 2.426.2, but not apparently triggered by upgrading (which was back in mid-December).

          The plugin that has raised my suspicions, given its install date:

          Jan 15 10:18 joda-time-api.jpi

          However, it is not giving me any option to downgrade it. Given the minimum Java requirement for Jenkins, I am unsure why plugins (pipeline-model-api, handy-uri-templates-2-api) are depending on it in the first place.

          Edit: I manually downgraded it to 2.12.5 but that did not help.

          James Howe added a comment - - edited I had it start while on 2.426.2, but not apparently triggered by upgrading (which was back in mid-December). The plugin that has raised my suspicions, given its install date: Jan 15 10:18 joda-time-api.jpi However, it is not giving me any option to downgrade it. Given the minimum Java requirement for Jenkins, I am unsure why plugins (pipeline-model-api, handy-uri-templates-2-api) are depending on it in the first place. Edit: I manually downgraded it to 2.12.5 but that did not help.

          James Howe added a comment - - edited

          I downgraded Jenkins to 2.426.1 and that does seems to have fixed the problem.

          Edit: turns out it didn't, it just got a lot less frequent

          James Howe added a comment - - edited I downgraded Jenkins to 2.426.1 and that does seems to have fixed the problem. Edit: turns out it didn't, it just got a lot less frequent

          James Howe added a comment -

          markewaite as Peter and I appear to have demonstrated that in this case it's a regression in 2.426.2, are you prepared to have a look for what the cause is? If you still think it's a local config issue, can you point us at where else we should be looking other than the system clock, locale, and service environment?

          James Howe added a comment - markewaite as Peter and I appear to have demonstrated that in this case it's a regression in 2.426.2, are you prepared to have a look for what the cause is? If you still think it's a local config issue, can you point us at where else we should be looking other than the system clock, locale, and service environment?

          Mark Waite added a comment -

          jameshowe I'm very interested in the problem. We've seen a similar failure on ci.jenkins.io. Unfortunately, I don't have any really strong ideas that would explain why the Jenkins controller would think that the system time has been set to 0 or to a small value that is near 0.

          Mark Waite added a comment - jameshowe I'm very interested in the problem. We've seen a similar failure on ci.jenkins.io . Unfortunately, I don't have any really strong ideas that would explain why the Jenkins controller would think that the system time has been set to 0 or to a small value that is near 0.

          James Howe added a comment - - edited

          My guess would be some failure in loading a job, causing it to be initialised with default values, and then written back out overwriting the original. It is possibly still caused by a plugin, but there's a difference in the error handling or the plugin interface that's only recently been introduced.

          While I was testing, I made backups of all the build.xml files, and replaced the broken ones after it happened. On the next restart that job was then back to normal.

          James Howe added a comment - - edited My guess would be some failure in loading a job, causing it to be initialised with default values, and then written back out overwriting the original. It is possibly still caused by a plugin, but there's a difference in the error handling or the plugin interface that's only recently been introduced. While I was testing, I made backups of all the build.xml files, and replaced the broken ones after it happened. On the next restart that job was then back to normal.

          ahmed added a comment -

          This issue is also happening to us, and sometimes new jobs are queued because of those old jobs with 1970 dates, I had to abort them using scripts as I couldn't from the UI.

           

          ahmed added a comment - This issue is also happening to us, and sometimes new jobs are queued because of those old jobs with 1970 dates, I had to abort them using scripts as I couldn't from the UI.  

          Pinned by James Howe

          Mark Waite added a comment -

          Can each of those that are affected by the problem please provide the detailed list of installed plugins and their versions?

          https://github.com/jenkins-infra/helpdesk/issues/3934#issuecomment-1932685147 indicates that the datadog plugin 6.0.0 may be the cause of some of the issues, though it is not yet clear if it is the cause of all of the issues.

          Mark Waite added a comment - Can each of those that are affected by the problem please provide the detailed list of installed plugins and their versions? https://github.com/jenkins-infra/helpdesk/issues/3934#issuecomment-1932685147 indicates that the datadog plugin 6.0.0 may be the cause of some of the issues, though it is not yet clear if it is the cause of all of the issues.

          James Howe added a comment -

          markewaite see my list in a previous comment. I'm afraid there's no Datadog plugin involved.

          James Howe added a comment - markewaite see my list in a previous comment. I'm afraid there's no Datadog plugin involved.

          Mark Waite added a comment - - edited

          jameshowe based on the list in your previous comment, I started an instance using those plugin versions. I don't see the problem with my local installation, but there are items in the list of plugins that are worth removing.

          The multiple SCMs plugin was deprecated 3 years ago. It now generates a stack trace in the console log when creating a new job. If your installation is still using it, then now is a good time to replace those jobs with Pipeline jobs.

          The code coverage plugin was deprecated a few months ago. The Warnings NG plugin provides a much better experience for code coverage users and is actively maintained and updated.

          The Tap plugin has a known security vulnerability. If you're not actively using it, then removing it is worth considering.

          I don't know how to duplicate the problem. Are there stack traces in the Jenkins controller log files that might guide further research. The ci.jenkins.io issue was narrowed to the datadog plugin based on contents of the controller log file. Maybe something similar will help in your case?

          Changes in Jenkins core between 2.426.1 and 2.426.2 were very few. They included:

          Mark Waite added a comment - - edited jameshowe based on the list in your previous comment, I started an instance using those plugin versions. I don't see the problem with my local installation, but there are items in the list of plugins that are worth removing. The multiple SCMs plugin was deprecated 3 years ago. It now generates a stack trace in the console log when creating a new job. If your installation is still using it, then now is a good time to replace those jobs with Pipeline jobs. The code coverage plugin was deprecated a few months ago. The Warnings NG plugin provides a much better experience for code coverage users and is actively maintained and updated. The Tap plugin has a known security vulnerability. If you're not actively using it, then removing it is worth considering. I don't know how to duplicate the problem. Are there stack traces in the Jenkins controller log files that might guide further research. The ci.jenkins.io issue was narrowed to the datadog plugin based on contents of the controller log file. Maybe something similar will help in your case? Changes in Jenkins core between 2.426.1 and 2.426.2 were very few. They included: Upgrade Winstone from 6.14 to 6.16 (upgrade Jetty from 10.0.17 to 10.0.18) Warn administrators of Java end of support 12 months and 3 months prior

          James Howe added a comment - - edited

          markewaite I cannot remove multiple-scms, because bitbucket-build-status-notifier still depends on it.
          code-coverage-api needs to remain in order to rebuild any older jobs that were using it, as the new coverage plugin is not backward-compatible.
          We are using the TAP plugin, and trust everyone with write access not to put injections in their test names etc.

          I am fully aware of all of these, because there are big notifications about them in Jenkins, including the one about CVE fixes in the latest Jenkins core, but unfortunately that triggers this issue.

          Looking at logs, there is a warning on startup for a stacktrace, but none of the affected projects use GitHub so I hadn't thought to check it. Will do some testing around there now: https://gist.github.com/OrangeDog/39fcde20f581d999c6dfe8a977cf2921

          There is also this, but maybe it's a symptom rather than the cause:

          WARNING        o.j.p.workflow.job.WorkflowRun#checkouts: JENKINS-26761: no checkouts in <job that lost a build> #14
          INFO        h.triggers.SCMTrigger$Runner#run: SCM changes detected in <job that lost a build>. Triggering  #15

          James Howe added a comment - - edited markewaite I cannot remove multiple-scms, because bitbucket-build-status-notifier still depends on it. code-coverage-api needs to remain in order to rebuild any older jobs that were using it, as the new coverage plugin is not backward-compatible. We are using the TAP plugin, and trust everyone with write access not to put injections in their test names etc. I am fully aware of all of these, because there are big notifications about them in Jenkins, including the one about CVE fixes in the latest Jenkins core, but unfortunately that triggers this issue. Looking at logs, there is a warning on startup for a stacktrace, but none of the affected projects use GitHub so I hadn't thought to check it. Will do some testing around there now: https://gist.github.com/OrangeDog/39fcde20f581d999c6dfe8a977cf2921 There is also this, but maybe it's a symptom rather than the cause: WARNING        o.j.p.workflow.job.WorkflowRun#checkouts: JENKINS-26761 : no checkouts in <job that lost a build> #14 INFO        h.triggers.SCMTrigger$Runner#run: SCM changes detected in <job that lost a build>. Triggering  #15

          James Howe added a comment - - edited

          Annoying update: downgrading core did not fix it. It has just happened to me again.

          I will upgrade back to the latest version and resume plugin testing when I get a chance.

          James Howe added a comment - - edited Annoying update: downgrading core did not fix it. It has just happened to me again. I will upgrade back to the latest version and resume plugin testing when I get a chance.

          Jeffrey added a comment -

          Hi all, we meet the same issue after bumping Jenkins version to `2.426.3`. 

          May I ask if this issue is fixed in the latest LTS version "2.440.1"? is it safe to bump to it now?

          Thank you!

          Jeffrey added a comment - Hi all, we meet the same issue after bumping Jenkins version to `2.426.3`.  May I ask if this issue is fixed in the latest LTS version "2.440.1"? is it safe to bump to it now? Thank you!

          Mark Waite added a comment -

          Plugin upgrades resolved the issue on ci.jenkins.io . Others have reported the issue without the specific plugins that seemed to cause the issue there. Unfortunately, other reports have not yet isolated a root cause.

          Mark Waite added a comment - Plugin upgrades resolved the issue on ci.jenkins.io . Others have reported the issue without the specific plugins that seemed to cause the issue there. Unfortunately, other reports have not yet isolated a root cause.

          James Howe added a comment -

          For me it still happens in 2.440.1. The GitHub and ShiningPanda plugins are not responsible. I've not yet investigated further.

          James Howe added a comment - For me it still happens in 2.440.1. The GitHub and ShiningPanda plugins are not responsible. I've not yet investigated further.

          Mark Waite added a comment - - edited

          jeffrey_cai those who might be interested to investigate the problem (like me) are not able to duplicate the problem. I've tried with the list of plugins provided by jameshowe and don't see the issue on my local computer. The issue that we were seeing on ci.jenkins.io was due to stack overflow issues in a release of the datadog plugin. The issue is fixed in the datadog plugin. However, jameshowe does not have the datadog plugin installed and yet still sees the issue.

          Can you duplicate the issue on a fresh Jenkins installation? Can you provide the list of plugins and their versions as reported by the system groovy script? Have you reviewed the system log for your Jenkins instance to see if there are stack traces or other hints in the system logs that might guide your investigation?

          Mark Waite added a comment - - edited jeffrey_cai those who might be interested to investigate the problem (like me) are not able to duplicate the problem. I've tried with the list of plugins provided by jameshowe and don't see the issue on my local computer. The issue that we were seeing on ci.jenkins.io was due to stack overflow issues in a release of the datadog plugin. The issue is fixed in the datadog plugin. However, jameshowe does not have the datadog plugin installed and yet still sees the issue. Can you duplicate the issue on a fresh Jenkins installation? Can you provide the list of plugins and their versions as reported by the system groovy script ? Have you reviewed the system log for your Jenkins instance to see if there are stack traces or other hints in the system logs that might guide your investigation?

          Jeffrey added a comment - - edited

          Hi Mark, thank you for taking a look. Below are some details:

          Unfortunately, the error log at that moment is gone.

          When I hit this 1970 issue, I did two things:

          1. bump Jenkins version from `2.426.1` to `2.426.3`
          2. bump some jenkins plugins, see the list below
          -credentials:1311.vcf0a_900b_37c2
          +credentials:1319.v7eb_51b_3a_c97b_
          -datadog:5.6.2
          +datadog:6.0.0
          -durable-task:543.v262f6a_803410
          +durable-task:547.vd1ea_007d100c
          -github:1.37.3.1
          +github:1.38.0
          -junit:1256.v002534a_5f33e
          +junit:1259.v65ffcef24a_88
          -kubernetes:4179.v3b_88431df708
          +kubernetes:4186.v1d804571d5d4
          -kubernetes-client-api:6.9.2-239.ve49a_3f285167
          +kubernetes-client-api:6.10.0-240.v57880ce8b_0b_2
          -metrics:4.2.18-442.v02e107157925
          +metrics:4.2.21-449.v6960d7c54c69
          -okhttp-api:4.11.0-157.v6852a_a_fa_ec11
          +okhttp-api:4.11.0-172.vda_da_1feeb_c6e
          -script-security:1313.v7a_6067dc7087
          +script-security:1321.va_73c0795b_923
          -workflow-api:1283.v99c10937efcb_
          +workflow-api:1291.v51fd2a_625da_7
          -workflow-durable-task-step:1317.v5337e0c1fe28
          +workflow-durable-task-step:1327.ve57634fb_09ce  

           

           

          Jeffrey added a comment - - edited Hi Mark, thank you for taking a look. Below are some details: Unfortunately, the error log at that moment is gone. When I hit this 1970 issue, I did two things: bump Jenkins version from `2.426.1` to `2.426.3` bump some jenkins plugins, see the list below -credentials:1311.vcf0a_900b_37c2 +credentials:1319.v7eb_51b_3a_c97b_ -datadog:5.6.2 +datadog:6.0.0 -durable-task:543.v262f6a_803410 +durable-task:547.vd1ea_007d100c -github:1.37.3.1 +github:1.38.0 -junit:1256.v002534a_5f33e +junit:1259.v65ffcef24a_88 -kubernetes:4179.v3b_88431df708 +kubernetes:4186.v1d804571d5d4 -kubernetes-client-api:6.9.2-239.ve49a_3f285167 +kubernetes-client-api:6.10.0-240.v57880ce8b_0b_2 -metrics:4.2.18-442.v02e107157925 +metrics:4.2.21-449.v6960d7c54c69 -okhttp-api:4.11.0-157.v6852a_a_fa_ec11 +okhttp-api:4.11.0-172.vda_da_1feeb_c6e -script-security:1313.v7a_6067dc7087 +script-security:1321.va_73c0795b_923 -workflow-api:1283.v99c10937efcb_ +workflow-api:1291.v51fd2a_625da_7 -workflow-durable-task-step:1317.v5337e0c1fe28 +workflow-durable-task-step:1327.ve57634fb_09ce    

          Mark Waite added a comment - - edited

          +datadog:6.0.0

          That was the plugin version that caused issues for ci.jenkins.io. Upgrade to the most recent release of the datadog plugin (6.0.2 or later) to avoid the stack overflow issues that were seen on ci.jenkins.io

          The complete list of plugins is needed in order to duplicate your environment, but the diff already shows the datadog 6.0.0 plugin that has a known problem. You can read more details of that problem in infra help desk 3934 and in Datadog plugin issue 393 that is fixed by datadog plugin pull request 396. The issue is resolved in datadog plugin 6.0.2 and later.

          Another case is described in a CloudBees knowledge base article, but the scenario described in that article seems specific to an upgrade then downgrade then upgrade scenario with a very old Jenkins version. I don't think that any of the cases described here match that article.

          Mark Waite added a comment - - edited +datadog:6.0.0 That was the plugin version that caused issues for ci.jenkins.io. Upgrade to the most recent release of the datadog plugin (6.0.2 or later) to avoid the stack overflow issues that were seen on ci.jenkins.io The complete list of plugins is needed in order to duplicate your environment, but the diff already shows the datadog 6.0.0 plugin that has a known problem. You can read more details of that problem in infra help desk 3934 and in Datadog plugin issue 393 that is fixed by datadog plugin pull request 396 . The issue is resolved in datadog plugin 6.0.2 and later. Another case is described in a CloudBees knowledge base article , but the scenario described in that article seems specific to an upgrade then downgrade then upgrade scenario with a very old Jenkins version. I don't think that any of the cases described here match that article.

          James Howe added a comment -

          It happens to me unpredictably and apparently at random, and I have ~7000 builds it could happen to. Though when it does happen it's usually to a more recent build, but it has happened to older ones (from years ago) occasionally. That probably means it will be hard to reproduce on an empty installation.

          Just have to hope it is one of the plugins I had upgraded in that time window, or I'm never going to find it myself.

          James Howe added a comment - It happens to me unpredictably and apparently at random, and I have ~7000 builds it could happen to. Though when it does happen it's usually to a more recent build, but it has happened to older ones (from years ago) occasionally. That probably means it will be hard to reproduce on an empty installation. Just have to hope it is one of the plugins I had upgraded in that time window, or I'm never going to find it myself.

          James Howe added a comment -

          It has not happened the last three times I've installed updates. Possibly it has been fixed (in Bitbucket Branch Source perhaps).

          James Howe added a comment - It has not happened the last three times I've installed updates. Possibly it has been fixed (in Bitbucket Branch Source perhaps).

          Mark Waite added a comment - - edited

          That's great to hear jameshowe. Shall we close this and reopen it if it reappears or leave it open for a week or two, then close it?

          Mark Waite added a comment - - edited That's great to hear jameshowe . Shall we close this and reopen it if it reappears or leave it open for a week or two, then close it?

          James Howe added a comment -

          At risk of jinxing it, it continues to be fine, including through the 2.440.2 upgrade.

          James Howe added a comment - At risk of jinxing it, it continues to be fine, including through the 2.440.2 upgrade.

          James Howe added a comment - - edited

          At risk of jinxing it,

          Guess what? The very next restart (for plugin updates) it happened again.

          James Howe added a comment - - edited At risk of jinxing it, Guess what? The very next restart (for plugin updates) it happened again.

          Jeffrey added a comment -

          Hi jameshowe , could you tell us the details of your plugin updates? what's the version? especially what's your version of datadog plugin? Thank you

          Jeffrey added a comment - Hi jameshowe , could you tell us the details of your plugin updates? what's the version? especially what's your version of datadog plugin? Thank you

          Mark Waite added a comment -

          jeffrey_cai , James had stated earlier that he does not have the Datadog plugin installed.

          Mark Waite added a comment - jeffrey_cai , James had stated earlier that he does not have the Datadog plugin installed.

          James Howe added a comment -

          jeffrey_cai I have already provided that information. I install every update when it comes out. Since the last list I have removed the GitHub and the ShiningPanda plugins. I do not have the Datadog plugin.

          There is some race condition, possibly related to Bitbucket Branch, that corrupts jobs rarely and at random after a restart, and build.xml gets overwritten on disk with an "empty" version. I have to manually backup and restore them. You will probably need a large history of many built jobs in order to reproduce.

          James Howe added a comment - jeffrey_cai I have already provided that information. I install every update when it comes out. Since the last list I have removed the GitHub and the ShiningPanda plugins. I do not have the Datadog plugin. There is some race condition, possibly related to Bitbucket Branch, that corrupts jobs rarely and at random after a restart, and build.xml gets overwritten on disk with an "empty" version. I have to manually backup and restore them. You will probably need a large history of many built jobs in order to reproduce.

          Jeffrey added a comment -

          Hi jameshowe thanks for your info.. this issue is a headache. As you mentioned below

          The very next restart (for plugin updates) it happened again.

          May I ask if you have the plugin diff in that "restart"? So I can make a cross-check on our side

          Jeffrey added a comment - Hi jameshowe thanks for your info.. this issue is a headache. As you mentioned below The very next restart (for plugin updates) it happened again. May I ask if you have the plugin diff in that "restart"? So I can make a cross-check on our side

          James Howe added a comment -

          jeffrey_cai it's not the plugin changes in the restart that cause it. Please read previous comments. Restarting with no changes also triggers it, but not every time.

          But if you insist, it was lockable-resources, ldap, and json-path-api.

          James Howe added a comment - jeffrey_cai it's not the plugin changes in the restart that cause it. Please read previous comments. Restarting with no changes also triggers it, but not every time. But if you insist, it was lockable-resources , ldap , and json-path-api .

          Jeffrey added a comment -

          Restarting with no changes also triggers it

          sorry I missed this point before, thank you very much. let me check our side.

          Jeffrey added a comment - Restarting with no changes also triggers it sorry I missed this point before, thank you very much. let me check our side.

          Jeffrey added a comment -

          Hi James, we noticed this plugin "lockable-resources" in your list (which is also used by us). 

          From their release notes, we noticed something about timestamp (but not sure if it's related), see 

          May I ask what's your version of this plugin? is it before the above fixed-version? if it's older, do you think upgrading to this version could help?

          Jeffrey added a comment - Hi James, we noticed this plugin " lockable-resources " in your list (which is also used by us).  From their release notes, we noticed something about timestamp (but not sure if it's related), see  https://github.com/jenkinsci/lockable-resources-plugin/releases fix version: 1245.vb_05f8a_4e28db_, Set the "reserved" timestamp when stealing a lock as well ( #637 ) May I ask what's your version of this plugin? is it before the above fixed-version? if it's older, do you think upgrading to this version could help?

          James Howe added a comment -

          jeffrey_cai it would save us both time if you read the previous comments. All plugins are the latest version. That change item has nothing to do with the issue.

          James Howe added a comment - jeffrey_cai it would save us both time if you read the previous comments. All plugins are the latest version. That change item has nothing to do with the issue.

          Jeffrey added a comment - - edited

          Thank you James, and sorry for the duplicated question. I saw you were using "Lockable Resources plugin (lockable-resources): 1232.v512d6c434eb_d". I have a guess (only a guess) that maybe bumping to `1245.vb_05f8a_4e28db_` could help.

          I will update here later after more testing from our side. thanks again

          Jeffrey added a comment - - edited Thank you James, and sorry for the duplicated question. I saw you were using "Lockable Resources plugin (lockable-resources): 1232.v512d6c434eb_d". I have a guess (only a guess) that maybe bumping to `1245.vb_05f8a_4e28db_` could help. I will update here later after more testing from our side. thanks again

          Kate added a comment - - edited

          Hi all. Jumping in, because we've just came across this issue, too, and also at the updating plugins stage. I hope a bit of a longer description of what we've seen is alright.

          We updated the plugins, restarted Jenkins (version 2.456), and then found that some (or maybe even all?) jobs had an amount of their recent builds look to be in a "running state" and dated at 1970, as per the picture attached. We looked into a few of those builds and their logs, and they weren't actually running, the logs were normal and from their actual run. We did this on the 1st of May, and the affected builds happened to be from the 4th of April, around the same time of the day, onwards.
          The builds weren't affected at the time of the restart itself, but at the time of even slight attempt at accessing them, even the view page, i.e. for example `/view/staging` , etc. If I don't click onto the next view, I can check one of that other view's job's build's build.xml in the terminal and it's good. When the view is changed, the build.xml files for the April 4th and later builds get updated and the builds in UI look to be "running".

          The build.xml gets actually changed - I took a copy of a couple, and for example one turned from 291 lines to 57 lines. Within the <action> section, it looks like all the plugins lines get removed; the <execution> and <checkouts> sections get entirely deleted out of the file; and the basic info like QueueId, timestamp, startTime, result, duration, charset, etc, just become these 4:

            <queueId>-1</queueId>
            <timestamp>0</timestamp>
            <startTime>0</startTime>
            <duration>0</duration>

           

          After a bit of time or another restart, the affected builds turned into looking like a failed state and they stayed like that.

          In our case, though, the culprit was the Datadog plugin - https://github.com/jenkinsci/datadog-plugin - and upgrading it from 6.0.2 to 7.0.0.
          Initially we upgraded all plugins at once, the above issue happened, so we rolled back (we had a pre-upgrade snapshot), and then I did a little research into the plugins till I narrowed it down to a few names, which then I tried updating on a non-production instance one by one. The Datadog plugin was what caused the problem happen again.
          I did consider the Lockable Resources plugin that was also mentioned in the comments here, but neither version of it caused any issue.
          The Datadog changes between the 6.0.2 version of the plugin and the current: https://github.com/jenkinsci/datadog-plugin/compare/728ed26b1e7a360fc1fb8674b3cbf40088fd84b0..master
          One of the 7.0.0 version fixes is actually "Fix queue time calculation for builds and pipeline steps.", so it could be related. As of 6.0.3 in their pom.xml at <scm> section it also turns from version 6.0.2 to 3.0.0 which is a bit confusing. (This 3.0.0 seems like it always happens between versions... a bit strange, but okay?)
          The above bits could be the only culprits in our case, or it could be the above's interaction with another part that goes badly.

          Our solution will probably just be removing the Datadog plugin, because we don't really use it, in the past it just seemed like a good idea to have and then it never happened...

          Kate added a comment - - edited Hi all. Jumping in, because we've just came across this issue, too, and also at the updating plugins stage. I hope a bit of a longer description of what we've seen is alright. We updated the plugins, restarted Jenkins (version 2.456), and then found that some (or maybe even all?) jobs had an amount of their recent builds look to be in a "running state" and dated at 1970, as per the picture attached. We looked into a few of those builds and their logs, and they weren't actually running, the logs were normal and from their actual run. We did this on the 1st of May, and the affected builds happened to be from the 4th of April, around the same time of the day, onwards. The builds weren't affected at the time of the restart itself, but at the time of even slight attempt at accessing them, even the view page, i.e. for example ` /view/staging ` , etc. If I don't click onto the next view, I can check one of that other view's job's build's build.xml in the terminal and it's good. When the view is changed, the build.xml files for the April 4th and later builds get updated and the builds in UI look to be "running". The build.xml gets actually changed - I took a copy of a couple, and for example one turned from 291 lines to 57 lines. Within the <action> section, it looks like all the plugins lines get removed; the <execution> and <checkouts> sections get entirely deleted out of the file; and the basic info like QueueId, timestamp, startTime, result, duration, charset, etc, just become these 4: <queueId>-1</queueId>   <timestamp>0</timestamp>   <startTime>0</startTime>   <duration>0</duration>   After a bit of time or another restart, the affected builds turned into looking like a failed state and they stayed like that. In our case, though, the culprit was the Datadog plugin - https://github.com/jenkinsci/datadog-plugin - and upgrading it from 6.0.2 to 7.0.0. Initially we upgraded all plugins at once, the above issue happened, so we rolled back (we had a pre-upgrade snapshot), and then I did a little research into the plugins till I narrowed it down to a few names, which then I tried updating on a non-production instance one by one. The Datadog plugin was what caused the problem happen again. I did consider the Lockable Resources plugin that was also mentioned in the comments here, but neither version of it caused any issue. The Datadog changes between the 6.0.2 version of the plugin and the current: https://github.com/jenkinsci/datadog-plugin/compare/728ed26b1e7a360fc1fb8674b3cbf40088fd84b0..master One of the 7.0.0 version fixes is actually "Fix queue time calculation for builds and pipeline steps.", so it could be related. As of 6.0.3 in their pom.xml at <scm> section it also turns from version 6.0.2 to 3.0.0 which is a bit confusing. (This 3.0.0 seems like it always happens between versions... a bit strange, but okay?) The above bits could be the only culprits in our case, or it could be the above's interaction with another part that goes badly. Our solution will probably just be removing the Datadog plugin, because we don't really use it, in the past it just seemed like a good idea to have and then it never happened...

          Idan Bidani added a comment -

          thank you for sharing your troubleshooting details, we stumbled on this bug as well after upgrading Core 2.387.3 -> 2.440.3. We also upgraded about 90 plugins as part of that which I can provide.
          We isolated the issue to the plugin allure-plugin 2.30.1 which wasn't upgraded. We also tried installing the latest version which didn't help. We saw the bug on the builds that were using the plugin and didn't see the bug on builds that didn't use it on the same job. It seems like there is some incompatibility between 2.440.3 Jenkins core and some plugins messing with build.xml

          I wanted to share in case it could help reproduce or bisect the bug

          Idan Bidani added a comment - thank you for sharing your troubleshooting details, we stumbled on this bug as well after upgrading Core 2.387.3 -> 2.440.3. We also upgraded about 90 plugins as part of that which I can provide. We isolated the issue to the plugin allure-plugin 2.30.1 which wasn't upgraded. We also tried installing the latest version which didn't help. We saw the bug on the builds that were using the plugin and didn't see the bug on builds that didn't use it on the same job. It seems like there is some incompatibility between 2.440.3 Jenkins core and some plugins messing with build.xml I wanted to share in case it could help reproduce or bisect the bug

            Unassigned Unassigned
            kishoregp Kishore
            Votes:
            4 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated: