Blocker NOTE: I've added a reproducer test case, and this issue matches my experience from testing it locally when I see the rollback work, and then still re-upgrade to the tainted level after going away from it previously. But I'm still not 100% sure of the diagnosis, so keep this in mind below.
Problem statement
When an instance taints a given update level (after a failure to upgrade), then it correctly gets the previous non tainted UL.
But then, when asking the backend again, passing the previous UL as parameter (i.e. the good one, the one before the just tainted one), then the backend will still serve an answer to upgrade to the tainted level.
From a user standpoint, this will surface through the following behavior:
- Imagine we are on UL 50.
- UL51 is pushed
- Instance updates to UL51 and is unable to restart
- Instance informs the backend UL51 was a failure (aka taints it)
- Instance goes back to UL50 from UL51
- Instance updates to UL51, fails...
- Loop back to a few lines above, and so on. I.e. the instance will do UL50->UL51->UL50->UL51, etc.
Expected behavior
Once per-instance tainted, a given UL should never be proposed again to a given instance.
- is blocking
-
JENKINS-53273
Implement the Evergreen Automated Rollback system
-
- Closed
-
- links to
[JENKINS-53983] Per-instance tainting still serves the tainted level if you're "coming from" a lower level
| Assignee | Original: R. Tyler Croy [ rtyler ] | New: Baptiste Mathus [ batmat ] |
| Status | Original: Open [ 1 ] | New: In Progress [ 3 ] |
| Link |
New:
This issue is blocking |
| Description |
Original:
_NOTE: I've added a [reproducer test case|https://github.com/jenkins-infra/evergreen/pull/316/commits/bf5a61c5312642efecb3007cf61a8903d5e59c08], and this issue matches my experience from testing it locally when I see the rollback work, and then still re-upgrade to the tainted level after going away from it previously. But I'm still not 100% sure of the diagnosis, so keep this in mind below._ h3. Problem statement When an instance taints a given update level (after a failure to upgrade), then it correctly gets the previous non tainted UL. But then, when asking the backend again, passing the previous UL as parameter (i.e. the good one, the one *before* the just tainted one), then the backend will still serve an answer to upgrade to the tainted level. h3. Expected behavior Once _per-instance tainted_ a given UL should never be proposed again to a given instance. |
New:
_NOTE: I've added a [reproducer test case|https://github.com/jenkins-infra/evergreen/pull/316/commits/bf5a61c5312642efecb3007cf61a8903d5e59c08], and this issue matches my experience from testing it locally when I see the rollback work, and then still re-upgrade to the tainted level after going away from it previously. But I'm still not 100% sure of the diagnosis, so keep this in mind below._ h3. Problem statement When an instance taints a given update level (after a failure to upgrade), then it correctly gets the previous non tainted UL. But then, when asking the backend again, passing the previous UL as parameter (i.e. the good one, the one *before* the just tainted one), then the backend will still serve an answer to upgrade to the tainted level. h3. Expected behavior Once _per-instance tainted_, a given UL should never be proposed again to a given instance. |
| Description |
Original:
_NOTE: I've added a [reproducer test case|https://github.com/jenkins-infra/evergreen/pull/316/commits/bf5a61c5312642efecb3007cf61a8903d5e59c08], and this issue matches my experience from testing it locally when I see the rollback work, and then still re-upgrade to the tainted level after going away from it previously. But I'm still not 100% sure of the diagnosis, so keep this in mind below._ h3. Problem statement When an instance taints a given update level (after a failure to upgrade), then it correctly gets the previous non tainted UL. But then, when asking the backend again, passing the previous UL as parameter (i.e. the good one, the one *before* the just tainted one), then the backend will still serve an answer to upgrade to the tainted level. h3. Expected behavior Once _per-instance tainted_, a given UL should never be proposed again to a given instance. |
New:
_NOTE: I've added a [reproducer test case|https://github.com/jenkins-infra/evergreen/pull/316/commits/bf5a61c5312642efecb3007cf61a8903d5e59c08], and this issue matches my experience from testing it locally when I see the rollback work, and then still re-upgrade to the tainted level after going away from it previously. But I'm still not 100% sure of the diagnosis, so keep this in mind below._ h3. Problem statement When an instance taints a given update level (after a failure to upgrade), then it correctly gets the previous non tainted UL. But then, when asking the backend again, passing the previous UL as parameter (i.e. the good one, the one *before* the just tainted one), then the backend will still serve an answer to upgrade to the tainted level. From a user standpoint, this will surface through the following behavior: * Imagine we are on UL 50. * UL51 is pushed * Instance updates to UL51 and is unable to restart * Instance informs the backend UL51 was a failure (aka _taints_ it) * Instance goes back to UL50 from UL51 * Instance updates to UL51, fails... * Loop back to a few lines above, and so on h3. Expected behavior Once _per-instance tainted_, a given UL should never be proposed again to a given instance. |
| Description |
Original:
_NOTE: I've added a [reproducer test case|https://github.com/jenkins-infra/evergreen/pull/316/commits/bf5a61c5312642efecb3007cf61a8903d5e59c08], and this issue matches my experience from testing it locally when I see the rollback work, and then still re-upgrade to the tainted level after going away from it previously. But I'm still not 100% sure of the diagnosis, so keep this in mind below._ h3. Problem statement When an instance taints a given update level (after a failure to upgrade), then it correctly gets the previous non tainted UL. But then, when asking the backend again, passing the previous UL as parameter (i.e. the good one, the one *before* the just tainted one), then the backend will still serve an answer to upgrade to the tainted level. From a user standpoint, this will surface through the following behavior: * Imagine we are on UL 50. * UL51 is pushed * Instance updates to UL51 and is unable to restart * Instance informs the backend UL51 was a failure (aka _taints_ it) * Instance goes back to UL50 from UL51 * Instance updates to UL51, fails... * Loop back to a few lines above, and so on h3. Expected behavior Once _per-instance tainted_, a given UL should never be proposed again to a given instance. |
New:
_NOTE: I've added a [reproducer test case|https://github.com/jenkins-infra/evergreen/pull/316/commits/bf5a61c5312642efecb3007cf61a8903d5e59c08], and this issue matches my experience from testing it locally when I see the rollback work, and then still re-upgrade to the tainted level after going away from it previously. But I'm still not 100% sure of the diagnosis, so keep this in mind below._ h3. Problem statement When an instance taints a given update level (after a failure to upgrade), then it correctly gets the previous non tainted UL. But then, when asking the backend again, passing the previous UL as parameter (i.e. the good one, the one *before* the just tainted one), then the backend will still serve an answer to upgrade to the tainted level. From a user standpoint, this will surface through the following behavior: * Imagine we are on UL 50. * UL51 is pushed * Instance updates to UL51 and is unable to restart * Instance informs the backend UL51 was a failure (aka _taints_ it) * Instance goes back to UL50 from UL51 * Instance updates to UL51, fails... * Loop back to a few lines above, and so on. I.e. the instance will do UL50->UL51->UL50->UL51, etc. h3. Expected behavior Once _per-instance tainted_, a given UL should never be proposed again to a given instance. |
| Status | Original: In Progress [ 3 ] | New: In Review [ 10005 ] |
| Rank | New: Ranked higher |
| Remote Link | New: This issue links to "evergreen PR (Web Link)" [ 21913 ] |
| Status | Original: In Review [ 10005 ] | New: In Progress [ 3 ] |