[JENKINS-53273] Implement the Evergreen Automated Rollback system

Type: Epic
Resolution: Won't Do
Priority: Critical
Component/s: evergreen
Labels:
- evergreen

Epic Name:
Evergreen Automatic Rollback System
Similar Issues:
Powered by SuggestiMate

Show
Sprint:
Evergreen - Milestone 1, Evergreen - Milestone 2

The automated rollback system has been designed, but not implemented yet.

Related JEPs/documents:

JEP 302 for the Git based snapshotting system
JEP 306 for the healthchecking

Acceptance criteria

Set up an evergreen instance, it is at UL-X
publish a (broken) update level (example: adding a plugin without a required dependency) => instance upgrades to UL-X
after restart, evergreen client detects Jenkins cannot restart, and rolls back to UL-X.
Jenkins is accessible again
a correct UL is published, Jenkins can upgrade to it

- - Sort By Name
  - Sort By Date
  - Ascending
  - Descending
  - Thumbnails
  - List
  - Download All

linux wut.png
22 kB
2018-09-04 13:04

blocks

JENKINS-53324 Author a jenkins.io blog post announcing Milestone 1

Resolved

depends on

JENKINS-53983 Per-instance tainting still serves the tainted level if you're "coming from" a lower level

Resolved

JENKINS-53964 Drop uuid_checksum_uniq unique key constraint

Closed

JENKINS-53499 Improve tests to avoid race condition for updates during startup

Closed

is blocking

JENKINS-53419 Demo Jenkins Evergreen successfully at DevOps World Jenkins World in SF

Resolved

Baptiste Mathus added a comment - 2018-09-04 13:04

Started trying to find some library to manage a git repo. Until now, no luck:

https://www.npmjs.com/package/run-git-command => IIUC, uses ES6 import feature. Which fails in our build as it is.
http://www.nodegit.org/ seemed very nice, but cannot get it to work on my machine. And it's fun but their CI actually says the build is borked on Linux:

> nodegit@0.22.2 install /home/tiste/dev/github/jenkins-infra/evergreen/distribution/client/node_modules/nodegit
> node lifecycleScripts/preinstall && node lifecycleScripts/install

[nodegit] Running pre-install script
[nodegit] Configuring libssh2.
{ Error: Command failed: /home/tiste/dev/github/jenkins-infra/evergreen/distribution/client/node_modules/nodegit/vendor/libssh2/configure --with-libssl-prefix=/home/tiste/dev/github/jenkins-infra/evergreen/distribution/client/node_modules/nodegit/vendor/openssl/openssl
/home/tiste/dev/github/jenkins-infra/evergreen/distribution/client/node_modules/nodegit/vendor/libssh2/missing: Unknown `--is-lightweight' option
Try `/home/tiste/dev/github/jenkins-infra/evergreen/distribution/client/node_modules/nodegit/vendor/libssh2/missing --help' for more information
configure: WARNING: 'missing' script is too old or missing
configure: error: No crypto library found!
Try --with-libssl-prefix=PATH
 or --with-libgcrypt-prefix=PATH
 or --with-wincng on Windows

    at ChildProcess.exithandler (child_process.js:275:12)
    at emitTwo (events.js:126:13)
    at ChildProcess.emit (events.js:214:7)
    at maybeClose (internal/child_process.js:925:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:209:5)
  killed: false,
  code: 1,
  signal: null,
  cmd: '/home/tiste/dev/github/jenkins-infra/evergreen/distribution/client/node_modules/nodegit/vendor/libssh2/configure --with-libssl-prefix=/home/tiste/dev/github/jenkins-infra/evergreen/distribution/client/node_modules/nodegit/vendor/openssl/openssl' }
/home/tiste/dev/github/jenkins-infra/evergreen/distribution/client/node_modules/nodegit/vendor/libssh2/missing: Unknown `--is-lightweight' option
Try `/home/tiste/dev/github/jenkins-infra/evergreen/distribution/client/node_modules/nodegit/vendor/libssh2/missing --help' for more information
configure: WARNING: 'missing' script is too old or missing
configure: error: No crypto library found!
Try --with-libssl-prefix=PATH
 or --with-libgcrypt-prefix=PATH
 or --with-wincng on Windows

[nodegit] ERROR - Could not finish preinstall
{ Error: Command failed: /home/tiste/dev/github/jenkins-infra/evergreen/distribution/client/node_modules/nodegit/vendor/libssh2/configure --with-libssl-prefix=/home/tiste/dev/github/jenkins-infra/evergreen/distribution/client/node_modules/nodegit/vendor/openssl/openssl
/home/tiste/dev/github/jenkins-infra/evergreen/distribution/client/node_modules/nodegit/vendor/libssh2/missing: Unknown `--is-lightweight' option
Try `/home/tiste/dev/github/jenkins-infra/evergreen/distribution/client/node_modules/nodegit/vendor/libssh2/missing --help' for more information
configure: WARNING: 'missing' script is too old or missing
configure: error: No crypto library found!
Try --with-libssl-prefix=PATH
 or --with-libgcrypt-prefix=PATH
 or --with-wincng on Windows

    at ChildProcess.exithandler (child_process.js:275:12)
    at emitTwo (events.js:126:13)
    at ChildProcess.emit (events.js:214:7)
    at maybeClose (internal/child_process.js:925:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:209:5)
  killed: false,
  code: 1,
  signal: null,
  cmd: '/home/tiste/dev/github/jenkins-infra/evergreen/distribution/client/node_modules/nodegit/vendor/libssh2/configure --with-libssl-prefix=/home/tiste/dev/github/jenkins-infra/evergreen/distribution/client/node_modules/nodegit/vendor/openssl/openssl' }
npm WARN evergreen-

Baptiste Mathus added a comment - 2018-09-04 13:04 Started trying to find some library to manage a git repo. Until now, no luck: https://www.npmjs.com/package/run-git-command => IIUC, uses ES6 import feature. Which fails in our build as it is. http://www.nodegit.org/ seemed very nice, but cannot get it to work on my machine. And it's fun but their CI actually says the build is borked on Linux: > nodegit@0.22.2 install /home/tiste/dev/github/jenkins-infra/evergreen/distribution/client/node_modules/nodegit > node lifecycleScripts/preinstall && node lifecycleScripts/install [nodegit] Running pre-install script [nodegit] Configuring libssh2. { Error: Command failed: /home/tiste/dev/github/jenkins-infra/evergreen/distribution/client/node_modules/nodegit/vendor/libssh2/configure --with-libssl-prefix=/home/tiste/dev/github/jenkins-infra/evergreen/distribution/client/node_modules/nodegit/vendor/openssl/openssl /home/tiste/dev/github/jenkins-infra/evergreen/distribution/client/node_modules/nodegit/vendor/libssh2/missing: Unknown `--is-lightweight' option Try `/home/tiste/dev/github/jenkins-infra/evergreen/distribution/client/node_modules/nodegit/vendor/libssh2/missing --help' for more information configure: WARNING: 'missing' script is too old or missing configure: error: No crypto library found! Try --with-libssl-prefix=PATH or --with-libgcrypt-prefix=PATH or --with-wincng on Windows at ChildProcess.exithandler (child_process.js:275:12) at emitTwo (events.js:126:13) at ChildProcess.emit (events.js:214:7) at maybeClose (internal/child_process.js:925:16) at Process.ChildProcess._handle.onexit (internal/child_process.js:209:5) killed: false, code: 1, signal: null, cmd: '/home/tiste/dev/github/jenkins-infra/evergreen/distribution/client/node_modules/nodegit/vendor/libssh2/configure --with-libssl-prefix=/home/tiste/dev/github/jenkins-infra/evergreen/distribution/client/node_modules/nodegit/vendor/openssl/openssl' } /home/tiste/dev/github/jenkins-infra/evergreen/distribution/client/node_modules/nodegit/vendor/libssh2/missing: Unknown `--is-lightweight' option Try `/home/tiste/dev/github/jenkins-infra/evergreen/distribution/client/node_modules/nodegit/vendor/libssh2/missing --help' for more information configure: WARNING: 'missing' script is too old or missing configure: error: No crypto library found! Try --with-libssl-prefix=PATH or --with-libgcrypt-prefix=PATH or --with-wincng on Windows [nodegit] ERROR - Could not finish preinstall { Error: Command failed: /home/tiste/dev/github/jenkins-infra/evergreen/distribution/client/node_modules/nodegit/vendor/libssh2/configure --with-libssl-prefix=/home/tiste/dev/github/jenkins-infra/evergreen/distribution/client/node_modules/nodegit/vendor/openssl/openssl /home/tiste/dev/github/jenkins-infra/evergreen/distribution/client/node_modules/nodegit/vendor/libssh2/missing: Unknown `--is-lightweight' option Try `/home/tiste/dev/github/jenkins-infra/evergreen/distribution/client/node_modules/nodegit/vendor/libssh2/missing --help' for more information configure: WARNING: 'missing' script is too old or missing configure: error: No crypto library found! Try --with-libssl-prefix=PATH or --with-libgcrypt-prefix=PATH or --with-wincng on Windows at ChildProcess.exithandler (child_process.js:275:12) at emitTwo (events.js:126:13) at ChildProcess.emit (events.js:214:7) at maybeClose (internal/child_process.js:925:16) at Process.ChildProcess._handle.onexit (internal/child_process.js:209:5) killed: false, code: 1, signal: null, cmd: '/home/tiste/dev/github/jenkins-infra/evergreen/distribution/client/node_modules/nodegit/vendor/libssh2/configure --with-libssl-prefix=/home/tiste/dev/github/jenkins-infra/evergreen/distribution/client/node_modules/nodegit/vendor/openssl/openssl' } npm WARN evergreen-

Jenkins

Details

Description

Acceptance criteria

Attachments

Attachments

Issue Links

Activity

Collapse comment: Baptiste Mathus added a comment - 2018-09-04 13:04

Expand comment: Baptiste Mathus added a comment - 2018-09-04 13:04

People

Dates