Queue of pending build requests is lost in case of two consecutive Jenkins restarts in a row

This issue is archived. You can view it, but you can't modify it. Learn more

XMLWordPrintable

      ... maybe "only" due to starting Jenkins in quiet mode via init.d Groovy hook script (to avoid starting queued builds right after restart), because oftentimes during Jenkins updates I need more than one restart.

      Note: After the 1st restart all pending build requests are still there, i.e. that works like expected.

      However, after the 2nd restart they are lost:

      • The "queue.xml.bak" seems to survive (with the pending build requests), but is not picked up after the 2nd restart?!

       

      Here are some logs, but maybe not too obvious or helpful:

      2013  20220408 10:40:56  sudo systemctl restart jenkins
2029  20220408 10:59:13  sudo systemctl restart jenkins

      r.fuereder@jenkins2:/var/lib/jenkins$ ll
...
-rw-r--r--   1 jenkins jenkins     877 Apr  8 10:39 queue.xml.bak
-rw-r--r--   1 jenkins jenkins    1204 Apr  8 10:41 scriptApproval.xml
...

r.fuereder@jenkins2:/var/lib/jenkins$ cat queue.xml.bak
<?xml version='1.1' encoding='UTF-8'?>
<hudson.model.Queue_-State>
  <counter>197</counter>
  <items>
    ... // 1 ("sandbox test stage")

</hudson.model.Queue_-State>r.fuereder@jenkins2:/var/lib/jenkins$ ll queue.xml.bak
-rw-r--r-- 1 jenkins jenkins 877 Apr  8 10:39 queue.xml.bak

Apr 08 10:39:12 jenkins2.xortex.com systemd[1]: Stopping Jenkins Continuous Integration Server...
Apr 08 10:39:12 jenkins2.xortex.com jenkins[1967675]: 2022-04-08 08:39:12.239+0000 [id=34]        INFO        hudson.lifecycle.Lifecycle#onStatusUpdate: Stopping Jenkins
Apr 08 10:39:12 jenkins2.xortex.com jenkins[1967675]: 2022-04-08 08:39:12.240+0000 [id=28]        INFO        winstone.Logger#logInternal: JVM is terminating. Shutting down Jetty
Apr 08 10:39:13 jenkins2.xortex.com systemd[1]: jenkins.service: Succeeded.
Apr 08 10:39:13 jenkins2.xortex.com systemd[1]: Stopped Jenkins Continuous Integration Server.
Apr 08 10:39:13 jenkins2.xortex.com systemd[1]: Starting Jenkins Continuous Integration Server...
Apr 08 10:39:14 jenkins2.xortex.com jenkins[1986392]: Listening for transport dt_socket at address: 5005
r.fuereder@jenkins2:/var/lib/jenkins$ ll queue.xml.bak
-rw-r--r-- 1 jenkins jenkins 877 Apr  8 10:39 queue.xml.bak
r.fuereder@jenkins2:/var/lib/jenkins$ stat queue.xml.bak
  File: queue.xml.bak
  Size: 877             Blocks: 8          IO Block: 4096   regular file
Device: 811h/2065d      Inode: 2622711     Links: 1
Access: (0644/-rw-r--r--)  Uid: (10112/ jenkins)   Gid: (  121/ jenkins)
Access: 2022-04-08 10:52:14.208895414 +0200
Modify: 2022-04-08 10:39:13.292919765 +0200
Change: 2022-04-08 10:39:31.273471075 +0200
 Birth: -

      => manually triggered "sandbox test git" pipeline

      r.fuereder@jenkins2:/var/lib/jenkins$ ll
...
-rw-r--r--   1 jenkins jenkins    1762 Apr  8 10:57 queue.xml
-rw-r--r--   1 jenkins jenkins     877 Apr  8 10:39 queue.xml.bak
...
r.fuereder@jenkins2:/var/lib/jenkins$ cat queue.xml
<?xml version='1.1' encoding='UTF-8'?>
<hudson.model.Queue_-State>
  <counter>2</counter>
  <items>
    ... // 2 prod, auto triggered by post commit hook

</hudson.model.Queue_-State>r.fuereder@jenkins2:/var/lib/jenkins$ cat queue.xml.bak
<?xml version='1.1' encoding='UTF-8'?>
<hudson.model.Queue_-State>
  <counter>197</counter>
  <items>
    ... // 1 ("sandbox test stage")

      => where is the "sandbox test git" above!?

      => Now the 2nd restart

      r.fuereder@jenkins2:/var/lib/jenkins$ sudo systemctl restart jenkins

r.fuereder@jenkins2:/var/lib/jenkins$ ll queue.*
-rw-r--r-- 1 jenkins jenkins 2538 Apr  8 10:59 queue.xml.bak
r.fuereder@jenkins2:/var/lib/jenkins$ cat queue.xml.bak
<?xml version='1.1' encoding='UTF-8'?>
<hudson.model.Queue_-State>
  <counter>3</counter>
  <items>
    ... // all 3 (2 prod, auto triggered by post commit hook; "sandbox test git") BUT not "sandbox test stage"

      => Now cancelled shutdown, i.e. the quiet mode => pending build requests start running

      r.fuereder@jenkins2:/var/lib/jenkins$ ll queue.*
-rw-r--r-- 1 jenkins jenkins 2538 Apr  8 10:59 queue.xml.bak
r.fuereder@jenkins2:/var/lib/jenkins$ cat queue.xml.bak
<?xml version='1.1' encoding='UTF-8'?>
<hudson.model.Queue_-State>
  <counter>3</counter>
  <items>
    ... // all 3 (2 prod, auto triggered by post commit hook; "sandbox test git")
  
r.fuereder@jenkins2:/var/lib/jenkins$ ll queue.*
-rw-r--r-- 1 jenkins jenkins  130 Apr  8 11:03 queue.xml
-rw-r--r-- 1 jenkins jenkins 2538 Apr  8 10:59 queue.xml.bak

r.fuereder@jenkins2:/var/lib/jenkins$ cat queue.xml
<?xml version='1.1' encoding='UTF-8'?>
<hudson.model.Queue_-State>
  <counter>12</counter>
  <items/>
  
r.fuereder@jenkins2:/var/lib/jenkins$ cat queue.xml.bak
<?xml version='1.1' encoding='UTF-8'?>
<hudson.model.Queue_-State>
  <counter>3</counter>
  <items>
    ... // all 3 (2 prod, auto triggered by post commit hook; "sandbox test git")

      => And half an hour later (one of the 3 is still running, the others are finished) still the same:

      r.fuereder@jenkins2:/var/lib/jenkins$ ll queue.*
-rw-r--r-- 1 jenkins jenkins  130 Apr  8 11:33 queue.xml
-rw-r--r-- 1 jenkins jenkins 2538 Apr  8 10:59 queue.xml.bak

r.fuereder@jenkins2:/var/lib/jenkins$ cat queue.xml
<?xml version='1.1' encoding='UTF-8'?>
<hudson.model.Queue_-State>
  <counter>26</counter>
  <items/>
</hudson.model.Queue_-State>

r.fuereder@jenkins2:/var/lib/jenkins$ cat queue.xml.bak
<?xml version='1.1' encoding='UTF-8'?>
<hudson.model.Queue_-State>
  <counter>3</counter>
  <items>
    ... // all 3 (2 prod, auto triggered by post commit hook; "sandbox test git")

       

            Assignee:
            Unassigned
            Reporter:
            Reinhold Füreder
            Archiver:
            Jenkins Service Account

              Created:
              Updated:
              Resolved:
              Archived: